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Abstract 


Ko, 


Internet Small Computer System Interface (iSCSI) Extensions for 
Remote Direct Memory Access (RDMA) provides the RDMA data transfer 
capability to iSCSI by layering iSCSI on top of an RDMA-Capable 
Protocol, such as the iWARP protocol suite. An RDMA-Capable Protocol 
provides RDMA Read and Write services, which enable data to be 
transferred directly into SCSI I/O Buffers without intermediate data 
copies. This document describes the extensions to the iSCSI protocol 
to support RDMA services as provided by an RDMA-Capable Protocol, 
such as the iWARP protocol suite. 
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Introduction 
Motivation 


The iSCSI protocol [RFC3720] is a mapping of the SCSI Architecture 


Model (see [SAM2]) over the TCP protocol. SCSI commands are carried 
by iSCSI requests, and SCSI responses and status are carried by iSCSI 
responses. Other iSCSI protocol exchanges and SCSI data are also 


transported in iSCSI Protocol Data Units (PDUs). 


Out-of-order TCP segments in the Traditional iSCSI model have to be 
stored and reassembled before the iSCSI protocol layer within an end 
node can place the data in the iSCSI buffers. This reassembly is 
required because not every TCP segment is likely to contain an iSCSI 
header to enable its placement, and TCP itself does not have a 
built-in mechanism for signaling Upper Level Protocol (ULP) message 
boundaries to aid placement of out-of-order segments. This TCP 
reassembly at high network speeds is quite counter-productive for the 
following reasons: wasted memory bandwidth in data copying, the need 
for reassembly memory, wasted CPU cycles in data copying, and the 
general store-and-forward latency from an application perspective. 
TCP reassembly was recognized as a serious issue in [RFC3720], and 
the notion of a "sync and steering layer" was introduced that is 
optional to implement and use. One specific sync and steering 
mechanism, called "markers", was defined in [RFC3720], which provides 
an application-level way of framing iSCSI Protocol Data Units (PDUs) 
within the TCP data stream even when the TCP segments are not yet 
reassembled to be in-order. 


With these defined techniques in [RFC3720], a Network Interface 
Controller customized for iSCSI (SNIC) could offload the TCP/IP 
processing and support direct data placement, but most iSCSI 
implementations do not support iSCSI "markers", making SNIC marker- 
based direct data placement unusable in practice. 


The iWARP protocol stack provides direct data placement functionality 
that is usable in practice. In addition, there is interest in using 
iSCSI with other Remote Direct Memory Access (RDMA) protocol stacks 
that support direct data placement, such as the one provided by 
InfiniBand. The generic term RDMA-Capable Protocol (RCaP) is used to 
refer to the RDMA functionality provided by such protocol stacks. 


With the availability of RDMA-Capable Controllers within a host 
system, which does not have SNICs, it is appropriate for iSCSI to be 
able to exploit the direct data placement function of the RDMA- 
Capable Controller like other applications. 
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iSCSI Extensions for RDMA (iSER) is designed precisely to take 
advantage of generic RDMA technologies -- iSER's goal is to permit 
iSCSI to employ direct data placement and RDMA capabilities using a 
generic RDMA-Capable Controller. In summary, the iSCSI/iSER protocol 
Stack is designed to enable scaling to high speeds by relying on a 
generic data placement process and RDMA technologies and products, 
which enable direct data placement of both in-order and out-of-order 
data. 


This document describes iSER as a protocol extension to iSCSI, both 
for convenience of description and because it is true in a very 
strict protocol sense. However, note that iSER is in reality 
extending the connectivity of the iSCSI protocol defined in 
[RFC3720], and the name iSER reflects this reality. 


When the iSCSI protocol as defined in [RFC3720] (i.e., without the 
iSER enhancements) is intended in the rest of the document, the term 
"Traditional iSCSI" is used to make the intention clear. 


1.2. Architectural Goals 


This section summarizes the architectural goals that guided the 
design of iSER. 


1. Provide an RDMA data transfer model for iSCSI that enables direct 
in-order or out-of-order data placement of SCSI data into pre- 
allocated SCSI buffers while maintaining in-order data delivery. 


2. Not require any major changes to the SCSI Architecture Model 
[SAM2] and SCSI command set standards. 


3. Utilize existing iSCSI infrastructure (sometimes referred to as 
"iSCSI ecosystem") including but not limited to MIB, 
bootstrapping, negotiation, naming and discovery, and security. 


4. Require a session to operate in the Traditional iSCSI data 
transfer mode if iSER is not supported by either the initiator or 
the target (i.e., not require iSCSI Full Feature Phase 
interoperability between an end node operating in Traditional 
iSCSI mode, and an end node operating in iSER-assisted mode). 


5. Allow initiator and target implementations to utilize generic 
RDMA-Capable Controllers such as RDMA-enabled Network Interface 
Controllers (RNICs), or to implement iSCSI and iSER in software 
(not require iSCSI- or iSER-specific assists in the RCaP 
implementation or RDMA-Capable Controller). 
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6. Require full and only generic RCaP functionality at both the 
initiator and the target. 


7. Implement a lightweight Datamover protocol for iSCSI with minimal 
state maintenance. 


1.3. Protocol Overview 


Consistent with the architectural goals stated in Section 2.2, the 
iSER protocol does not require changes in the iSCSI ecosystem or any 
related SCSI specifications. The iSER protocol defines the mapping 
of iSCSI PDUs to RCaP Messages in such a way that it is entirely 
feasible to realize iSCSI/iSER implementations that are based on 
generic RDMA-Capable Controllers. The iSER protocol layer requires 
minimal state maintenance to assist an iSCSI Full Feature Phase 
connection, besides being oblivious to the notion of an iSCSI 
session. The crucial protocol aspects of iSER may be summarized 
thus: 


1. iSER-assisted mode is negotiated during the iSCSI login for each 
session, and an entire iSCSI session can only operate in one mode 
(i.e., a connection in a session cannot operate in iSER-assisted 
mode if a different connection of the same session is already in 
Full Feature Phase in the Traditional iSCSI mode). 


2. Once in iSER-assisted mode, all iSCSI interactions on that 
connection use RCaP Messages. 


3. A Send Message Type is used for carrying an iSCSI control-type PDU 
preceded by an iSER header. See Section 7.2 for more details on 
iSCSI control-type PDUs. 


4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages are 
used for carrying control and all data information associated with 
the iSCSI data-type PDUs. See Section 7.1 for more details on 
iSCSI data-type PDUs. 


5. Target drives all data transfer (with the exception of iSCSI 
unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA 
Read Requests and RDMA Writes, respectively. 


6. RCaP is responsible for ensuring data integrity. (For example, 
iWARP includes a CRC-enhanced framing layer called Marker PDU 
Aligned Framing for TCP (MPA) on top of TCP; and for InfiniBand, 
the CRCs are included in the Reliable Connection mode). For this 
reason, iSCSI header and data digests are negotiated to "None" for 
iSCSI/iSER sessions. 
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7. The iSCSI error recovery hierarchy defined in [RFC3720] is fully 
supported by iSER. (However, see Section 7.3.11 on the handling 
of SNACK Request PDUs.) 


8. iSER requires no changes to iSCSI authentication, security, and 
text mode negotiation mechanisms. 


Note that Traditional iSCSI implementations may have to be adapted to 
employ iSER. It is expected that the adaptation when required is 
likely to be centered around the upper layer interface requirements 
of iSER (Section 3). 


RDMA Services and iSER 


iSER is designed to work with software and/or hardware protocol 
stacks providing the protocol services defined in RCaP documents such 
as [RDMAP], [IB], etc. The following subsections describe the key 
protocol elements of RCaP services that iSER relies on. 


l. STag 
A Steering Tag (STag) is the identifier of an I/O Buffer unique to an 


RDMA-Capable Controller that the iSER layer Advertises to the remote 
iSCSI/iSER node in order to complete a SCSI I/O. 


In iSER, Advertisement is the act of informing the target by the 
initiator that an I/O Buffer is available at the initiator for RDMA 
Read or RDMA Write access by the target. The initiator Advertises 
the I/O Buffer by including the STag in the header of an iSER Message 
containing the SCSI Command PDU to the target. The base Tagged 
Offset is not explicitly specified, but the target must always assume 
it as zero. The buffer length is as specified in the SCSI Command 
PDU. 


The iSER layer at the initiator Advertises the STag for the I/O 
Buffer of each SCSI I/O to the iSER layer at the target in the iSER 
header of the Send with Solicited Event (SendSE) Message containing 
the SCSI Command PDU, unless the I/O can be completely satisfied by 
unsolicited data alone. 


The iSER layer at the target provides the STag for the I/O Buffer 
that is the Data Sink of an RDMA Read Operation (Section 2.4.4) to 
the RCaP layer on the initiator node -- i.e., this is completely 
transparent to the iSER layer at the initiator. 


The iSER protocol is defined so that the Advertised STag is 


automatically invalidated upon a normal completion of the associated 
task. This automatic invalidation is realized via the Send with 
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Solicited Event and Invalidate (SendInvSE) Message carrying the SCSI 
Response PDU. There are two exceptions to this automatic 
invalidation -- bidirectional commands, and abnormal completion of a 
command. The iSER layer at the initiator is required to explicitly 
invalidate the STag in these cases, in addition to sanity checking 
the automatic invalidation even when that does happen. 


.2. Send 


Send is the RDMA Operation that is not addressed to an Advertised 
buffer by the sending side, and thus uses Untagged buffers on the 
receiving side. 


The iSER layer at the initiator uses the Send Operation to transmit 
any iSCSI control-type PDU to the target. As an example, the 
initiator uses Send Operations to transfer iSER Messages containing 
SCSI Command PDUs to the iSER layer at the target. 


An iSER layer at the target uses the Send Operation to transmit any 
iSCSI control-type PDU to the initiator. As an example, the target 
uses Send Operations to transfer iSER Messages containing SCSI 
Response PDUs to the iSER layer at the initiator. 


3. RDMA Write 


RDMA Write is the RDMA Operation that is used to place data into an 
Advertised buffer on the receiving side. The sending side addresses 
the Message using an STag and a Tagged Offset that are valid on the 
Data Sink. 


The iSER layer at the target uses the RDMA Write Operation to 
transfer the contents of a local I/O Buffer to an Advertised I/O 
Buffer at the initiator. The iSER layer at the target uses the RDMA 
Write to transfer whole or part of the data required to complete a 
SCSI read command. 


The iSER layer at the initiator does not employ RDMA Writes. 
4. RDMA Read 


RDMA Read is the RDMA Operation that is used to retrieve data from an 
Advertised buffer on a remote node. The sending side of the RDMA 
Read Request addresses the Message using an STag and a Tagged Offset 
that are valid on the Data Source in addition to providing a valid 
local STag and Tagged Offset that identify the Data Sink. 


The iSER layer at the target uses the RDMA Read Operation to transfer 
the contents of an Advertised I/O Buffer at the initiator to a local 
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I/O Buffer at the target. The iSER layer at the target uses the RDMA 
Read to fetch whole or part of the data required to complete a SCSI 
write command. 


The iSER layer at the initiator does not employ RDMA Reads. 
1.5. SCSI Read Overview 


The iSER layer at the initiator receives the SCSI Command PDU from 
the iSCSI layer. The iSER layer at the initiator generates an STag 
for the I/O Buffer of the SCSI Read and Advertises the buffer by 
including the STag as part of the iSER header for the PDU. The iSER 
Message is transferred to the target using a SendSE Message. 


The iSER layer at the target uses one or more RDMA Writes to transfer 
the data required to complete the SCSI Read. 


The iSER layer at the target uses a SendInvSE Message to transfer the 
SCSI Response PDU back to the iSER layer at the initiator. The iSER 
layer at the initiator notifies the iSCSI layer of the availability 
of the SCSI Response PDU. 


1.6. SCSI Write Overview 


The iSER layer at the initiator receives the SCSI Command PDU from 
the iSCSI layer. If solicited data transfer is involved, the iSER 
layer at the initiator generates an STag for the I/O Buffer of the 
SCSI Write and Advertises the buffer by including the STag as part of 
the iSER header for the PDU. The iSER Message is transferred to the 
target using a SendSE Message. 


The iSER layer at the initiator may optionally send one or more non- 
immediate unsolicited data PDUs to the target using Send Message 
Types. 


If solicited data transfer is involved, the iSER layer at the target 
uses one or more RDMA Reads to transfer the data required to complete 
the SCSI Write. 


The iSER layer at the target uses a SendInvSE Message to transfer the 
SCSI Response PDU back to the iSER layer at the initiator. The iSER 
layer at the initiator notifies the iSCSI layer of the availability 
of the SCSI Response PDU. 

1.7.  iSCSI/iSER Layering 


iSCSI Extensions for RDMA (iSER) is layered between the iSCSI layer 
and the RCaP layer. Note that the RCaP layer may be composed of one 
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or more distinct protocol layers depending on the specifics of the 
RCaP. Figure 1 shows an example of the relationship between SCSI, 
iSCSI, iSER, and the different RCaP layers. For TCP, the RCaP is 
iWARP. For InfiniBand, the RCaP is the Reliable Connected Transport 
Service. Note that the iSCSI layer as described here supports the 
RDMA Extensions as used in iSER. 


4------------------------------------- * 
| SCSI | 
4------------------------------------- * 
| iSCSI | 
DI ------ AR * 
| iSER | 
下 站 生生 全 fort einan + 
| RDMAP | | 
Passes ous + InfiniBand 
| DDE | Reliable Other 
下 + Connected | RDMA- | 
| MPA | Transport | Capable | 
T--------- 十 Service | Protocol | 
| Tc | | | 
本 二 二 4-------------- 4R------------ * 
| | InfiniBand | Other | 
| IP | Network | Network | 
| | Layer | Layer | 
es 二 二 车 二 二 十 三 三 二 二 一 二 二 二 后 一 全 二 十 


Figure 1. Example of iSCSI/iSER Layering in Full Feature Phase 
2. Definitions and Acronyms 
2.1. Definitions 


Advertisement (Advertised, Advertise, Advertisements, Advertises) 一 
The act of informing a remote iSER layer that a local node’s 
buffer is available to it. A Node makes a buffer available for 
incoming RDMA Read Request Message or incoming RDMA Write Message 
access by informing the remote iSER layer of the Tagged Buffer 
identifiers (STag, TO, and buffer length). Note that this 
Advertisement of Tagged Buffer information is the responsibility 
of the iSER layer on either end and is not defined by the RDMA- 
Capable Protocol. A typical method would be for the iSER layer to 
embed the Tagged Buffer’s STag, TO, and buffer length in a Send 
Message destined for the remote iSER layer. 


Completion (Completed, Complete, Completes) - Completion is defined 
as the process by the RDMA-Capable Protocol layer to inform the 
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iSER layer, that a particular RDMA Operation has performed all 
functions specified for the RDMA Operation. 


Connection - A connection is a logical circuit between the initiator 
and the target, e.g., a TCP connection. Communication between the 
initiator and the target occurs over one or more connections. The 
connections carry control messages, SCSI commands, parameters, and 
data within iSCSI Protocol Data Units (iSCSI PDUs). 


Connection Handle - An information element that identifies the 
particular iSCSI connection and is unique for a given iSCSI-iSER 
pair. Every invocation of an Operational Primitive is qualified 
with the Connection Handle. 


Data Sink - The peer receiving a data payload. Note that the Data 
Sink can be required to both send and receive RCaP Messages to 
transfer a data payload. 


Data Source - The peer sending a data payload. Note that the Data 
Source can be required to both send and receive RCaP Messages to 
transfer a data payload. 


Datamover Interface (DI) - The interface between the iSCSI layer and 
the Datamover layer as described in [DA]. 


Datamover Layer - A layer that is directly below the iSCSI layer and 
above the underlying transport layers. This layer exposes and 
uses a set of transport independent Operational Primitives for the 
communication between the iSCSI layer and itself. The Datamover 
layer, operating in conjunction with the transport layers, moves 
the control and data information on the iSCSI connection. In this 
Specification, the iSER layer is the Datamover layer. 


Datamover Protocol - A Datamover protocol is the wire-protocol that 
is defined to realize the Datamover layer functionality. In this 
Specification, the iSER protocol is the Datamover protocol. 


Event - An indication provided by the RDMA-Capable Protocol layer to 
the iSER layer to indicate a Completion or other condition 
requiring immediate attention. 


Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming 
outstanding RDMA Read Requests that the RDMA-Capable Controller 
can handle on a particular RCaP Stream at the Data Source. For 
some RDMA-Capable Protocol layers, the term "IRD" may be known by 
a different name. For example, for InfiniBand, the equivalent for 
IRD is the Responder Resources. 
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Invalidate STag - A mechanism used to prevent the Remote Peer from 
reusing a previous explicitly Advertised STag, until the iSER 
layer at the local node makes it available through a subsequent 
explicit Advertisement. 


I/O Buffer - A buffer that is used in a SCSI Read or Write operation 
So SCSI data may be sent from or received into that buffer. 


iSCSI - The iSCSI protocol as defined in [RFC3720] is a mapping of 
the SCSI Architecture Model of SAM-2 over TCP. 


iSCSI control-type PDU - Any iSCSI PDU that is not an iSCSI data- 
type PDU and also not a SCSI Data-out PDU carrying solicited data 
is defined as an iSCSI control-type PDU. Specifically, it is to 
be noted that SCSI Data-out PDUs for unsolicited data are defined 
as iSCSI control-type PDUs. 


iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI 

PDU that causes data transfer, transparent to the remote iSCSI 
layer, to take place between the peer iSCSI nodes on a Full 
Feature Phase iSCSI connection. An iSCSI data-type PDU, when 
requested for transmission by the sender iSCSI layer, results in 
the associated data transfer without the participation of the 
remote iSCSI layer, i.e. the PDU itself is not delivered as-is to 
the remote iSCSI layer. The following iSCSI PDUs constitute the 
set of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU. 


iSCSI Layer - A layer in the protocol stack implementation within an 
end node that implements the iSCSI protocol and interfaces with 
the iSER layer via the Datamover Interface. 

iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI layer at the 
initiator and the iSCSI layer at the target divide their 
communications into messages. The term "iSCSI protocol data unit" 
(iSCSI PDU) is used for these messages. 

iSCSI/iSER Connection - An iSER-assisted iSCSI connection. 

iSCSI/iSER Session - An iSER-assisted iSCSI session. 

iSCSI-iSER Pair - The iSCSI layer and the underlying iSER layer. 


iSER - iSCSI Extensions for RDMA, the protocol defined in this 
document. 


iSER-assisted - A term generally used to describe the operation of 


iSCSI when the iSER functionality is also enabled below the iSCSI 
layer for the specific iSCSI/iSER connection in question. 
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iSER-IRD - This variable represents the maximum number of incoming 
outstanding RDMA Read Requests that the iSER layer at the 
initiator declares on a particular RCaP Stream. 


iSER-ORD - This variable represents the maximum number of outstanding 
RDMA Read Requests that the iSER layer can initiate on a 
particular RCaP Stream. This variable is maintained only by the 
iSER layer at the target. 


iSER Layer - The layer that implements the iSCSI Extensions for RDMA 
(iSER) protocol. 


iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and 
[MPA] when layered above [TCP]. [RDMAP] and [DDP] may be layered 
above SCTP or other transport protocols. 


Local Mapping - A task state record maintained by the iSER layer that 
associates the Initiator Task Tag to the local STag(s). The 
specifics of the record structure are implementation dependent. 


Local Peer - The implementation of the RDMA-Capable Protocol on the 
local end of the connection. Used to refer to the local entity 
when describing protocol exchanges or other interactions between 
two Nodes. 


Node - A computing device attached to one or more links of a network. 
A Node in this context does not refer to a specific application or 
protocol instantiation running on the computer. A Node may 
consist of one or more RDMA-Capable Controllers installed in a 
host computer. 


Operational Primitive - An Operational Primitive is an abstract 
functional interface procedure that requests that another layer 
perform a specific action on the requestor's behalf or notifies 
the other layer of some event. The Datamover Interface between an 
iSCSI layer and a Datamover layer within an iSCSI end node uses a 
set of Operational Primitives to define the functional interface 
between the two layers. Note that not every invocation of an 
Operational Primitive may elicit a response from the requested 
layer. A full discussion of the Operational Primitive types and 
request-response semantics available to iSCSI and iSER can be 
found in [DA]. 


Outbound RDMA Read Queue Depth (ORD) - The maximum number of 
outstanding RDMA Read Requests that the RDMA-Capable Controller 
can initiate on a particular RCaP Stream at the Data Sink. 了 or 
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some RDMA-Capable Protocol layer, the term "ORD" may be known by a 
different name. For example, for InfiniBand, the equivalent for 
ORD is the Initiator Depth. 


Phase-Collapse - Refers to the optimization in iSCSI where the SCSI 
status is transferred along with the final SCSI Data-in PDU froma 
target. See Section 3.2 in [RFC3720]. 


RCaP Message - One or more packets of the network layer comprising a 
single RDMA Operation or a part of an RDMA Read Operation of the 
RDMA-Capable Protocol. For iWARP, an RCaP Message is known as an 
RDMAP Message. 


RCaP Stream - A single bidirectional association between the peer 
RDMA-Capable Protocol layers on two Nodes over a single 
transport-level stream. For iWARP, an RCaP Stream is known as an 
RDMAP Stream, and the association is created when the connection 
transitions to iSER-assisted mode following a successful Login 
Phase during which iSER support is negotiated. 


RDMA-Capable Protocol (RCaP) - The protocol or protocol suite that 
provides a reliable RDMA transport functionality, e.g., iWARP, 
InfiniBand, etc. 


RDMA-Capable Controller - A network I/O adapter or embedded 
controller with RDMA functionality. For example, for iWARP, this 
could be an RNIC, and for InfiniBand, this could be a HCA (Host 
Channel Adapter) or TCA (Target Channel Adapter). 


RDMA-enabled Network Interface Controller (RNIC) - A network I/O 
adapter or embedded controller with iWARP functionality. 


RDMA Operation - A sequence of RCaP Messages, including control 
Messages, to transfer data from a Data Source to a Data Sink. The 
following RDMA Operations are defined - RDMA Write Operation, RDMA 
Read Operation, Send Operation, Send with Invalidate Operation, 
Send with Solicited Event Operation, Send with Solicited Event and 
Invalidate Operation, and Terminate Operation. 


RDMA Protocol (RDMAP) - A wire protocol that supports RDMA Operations 
to transfer ULP data between a Local Peer and the Remote Peer as 
described in [RDMAP]. 


RDMA Read Operation - An RDMA Operation used by the Data Sink to 
transfer the contents of a Data Source buffer from the Remote Peer 
to a Data Sink buffer at the Local Peer. An RDMA Read operation 
consists of a single RDMA Read Request Message and a single RDMA 
Read Response Message. 
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RDMA Read Request - An RCaP Message used by the Data Sink to request 
that the Data Source transfer the contents of a buffer. The RDMA 
Read Request Message describes both the Data Source and the Data 
Sink buffers. 


RDMA Read Response - An RCaP Message used by the Data Source to 
transfer the contents of a buffer to the Data Sink, in response to 
an RDMA Read Request. The RDMA Read Response Message only 
describes the Data Sink buffer. 


RDMA Write Operation - An RDMA Operation used by the Data Source to 
transfer the contents of a Data Source buffer from the Local Peer 
to a Data Sink buffer at the Remote Peer. The RDMA Write Message 
only describes the Data Sink buffer. 


Remote Direct Memory Access (RDMA) - A method of accessing memory on 
a remote system in which the local system specifies the remote 
location of the data to be transferred.  Employing an RDMA- 
Capable Controller in the remote system allows the access to take 
place without interrupting the processing of the CPU(s) on the 
system. 


Remote Mapping - A task state record maintained by the iSER layer 
that associates the Initiator Task Tag to the Advertised STag(s). 
The specifics of the record structure are implementation 
dependent. 


Remote Peer - The implementation of the RDMA-Capable Protocol on the 
opposite end of the connection. Used to refer to the remote 
entity when describing protocol exchanges or other interactions 
between two Nodes. 


SCSI Layer - This layer builds/receives SCSI CDBs (Command Descriptor 
Blocks) and sends/receives them with the remaining command execute 
[SAM2] parameters to/from the iSCSI layer. 


Send - An RDMA Operation that transfers the contents of a Buffer from 
the Local Peer to a Buffer at the Remote Peer. 


Send Message Type - A Send Message, Send with Invalidate Message, 
Send with Solicited Event Message, or Send with Solicited Event 
and Invalidate Message. 


SendInvSE Message - A Send with Solicited Event and Invalidate 
Message. 


SendSE Message - A Send with Solicited Event Message. 
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Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for an 
R2T PDU. The semantics for both types of sequence numbers are as 
defined in [RFC3720]. 


Session, iSCSI Session - The group of connections that link an 
initiator SCSI port with a target SCSI port form an iSCSI session 
(equivalent to a SCSI I-T nexus). Connections can be added to and 
removed from a session even while the I-T nexus is intact. Across 
all connections within a session, an initiator sees one and the 
same target. 


Solicited Event (SE) - A facility by which an RDMA Operation sender 
may cause an Event to be generated at the recipient, if the 
recipient is configured to generate such an Event, when a Send 
with Solicited Event or Send with Solicited Event and Invalidate 
Message is received. 


Steering Tag (STag) - An identifier of a Tagged Buffer on a Node 
(Local or Remote) as defined in [RDMAP] and [DDP]. For other 
RDMA-Capable Protocols, the Steering Tag may be known by different 
names but will be herein referred to as STags. For example, for 
InfiniBand, a Remote STag is known as an R-Key, and a local STag 
is known as an L-Key, and both will be considered STags. 


Tagged Buffer - A buffer that is explicitly Advertised to the iSER 
layer at the remote node through the exchange of an STag, Tagged 
Offset, and length. 

Tagged Offset (TO) - The offset within a Tagged Buffer. 


Traditional iSCSI - Refers to the iSCSI protocol as defined in 
[RFC3720] (i.e. without the iSER enhancements). 


Untagged Buffer - A buffer that is not explicitly Advertised to the 
iSER layer at the remode node. 


2.2. Acronyms 


Acronym Definition 

AHS Additional Header Segment 
BHS Basic Header Segment 

CO Connection Only 

CRC Cyclic Redundancy Check 
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DDP Direct Data Placement Protocol 

DI Datamover Interface 

HCA Host Channel Adapter 

TANA Internet Assigned Numbers Authority 
IB InfiniBand 

IETF Internet Engineering Task Force 

I/O Input - Output 

IO Initialize Only 

IP Internet Protocol 

IPOIB IP over InfiniBand 

IPsec Internet Protocol Security 

iSER iSCSI Extensions for RDMA 

TET Initiator Task Tag 

LO Leading Only 

MPA Marker PDU Aligned Framing for TCP 
NOP No Operation 

NSG Next Stage (during the iSCSI Login Phase) 
OS Operating System 

PDU Protocol Data Unit 

R2T Ready To Transfer 

R2TSN Ready To Transfer Sequence Number 
RDMA Remote Direct Memory Access 

RDMAP Remote Direct Memory Access Protocol 
RFC Request For Comments 
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RNIC RDMA-enabled Network Interface Controller 
SAM2 SCSI Architecture Model - 2 

SCSI Small Computer Systems Interface 

SNACK Selective Negative Acknowledgment - also 


Sequence Number Acknowledgement for data 


STag Steering Tag 

SW Session Wide 

TCA Target Channel Adapter 

TCP Transmission Control Protocol 
TMF Task Management Function 

TTT Target Transfer Tag 

TO Tagged Offset 

ULP Upper Level Protocol 
Conventions 


The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 
document are to be interpreted as described in [RFC2119]. 


Upper Layer Interface Requirements 


This section discusses the upper layer interface requirements in the 
form of an abstract model of the required interactions between the 
iSCSI layer and the iSER layer. The abstract model used here is 
derived from the architectural model described in [DA]. [DA] also 
provides a functional overview of the interactions between the iSCSI 
layer and the Datamover layer as intended by the Datamover 
Architecture. 


The interface requirements are specified by Operational Primitives. 
An Operational Primitive is an abstract functional interface 
procedure between the iSCSI layer and the iSER layer that requests 
one layer to perform a specific action on behalf of the other layer 
or notifies the other layer of some event. Whenever an Operational 
Primitive in invoked, the Connection Handle qualifier is used to 
identify a particular iSCSI connection. For some Operational 
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Primitives, a Data Descriptor is used to identify the iSCSI/SCSI data 
buffer associated with the requested or completed operation. 


The abstract model and the Operational Primitives defined in this 
section facilitate the description of the iSER protocol. In the rest 
of the iSER specification, the compliance statements related to the 
use of these Operational Primitives are only for the purpose of the 
required interactions between the iSCSI layer and the iSER layer. 
Note that the compliance statements related to the Operational 
Primitives in the rest of this specification only mandate functional 
equivalence on implementations, but do not put any requirements on 
the implementation specifics of the interface between the iSCSI layer 
and the iSER layer. 


Each Operational Primitive is invoked with a set of qualifiers that 
specify the information context for performing the specific action 
being requested of the Operational Primitive. While the qualifiers 
are required, the method of realizing the qualifiers (e.g., by 
passing synchronously with invocation, or by retrieving from task 


context, or by retrieving from shared memory, etc.) is implementation 
dependent. 
Operational Primitives Offered by iSER 


The iSER protocol layer MUST support the following Operational 
Primitives to be used by the iSCSI protocol layer. 


1. Send_Control 


Input qualifiers: Connection Handle, BHS and AHS (if any) of the 
iSCSI PDU, PDU-specific qualifiers 


Return results: Not specified 


This is used by the iSCSI layers at the initiator and the target to 
request the outbound transfer of an iSCSI control-type PDU (see 
Section 7.2). Qualifiers that only apply for a particular control- 
type PDU are known as PDU-specific qualifiers, e.g., 
ImmediateDataSize for a SCSI write command. For details on PDU- 
specific qualifiers, see Section 7.3. The iSCSI layer can only 
invoke the Send_Control Operational Primitive when the connection is 
in iSER-assisted mode. 


.2. Put Data 


Input qualifiers: Connection Handle, content of a SCSI Data-in 
PDU header, Data Descriptor, Notify Enable 
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Return results: Not specified 


This is used by the iSCSI layer at the target to request the outbound 
transfer of data for a SCSI Data-in PDU from the buffer identified by 
the Data Descriptor qualifier. The iSCSI layer can only invoke the 
Put Data Operational Primitive when the connection is in iSER- 
assisted mode. 


The Notify Enable qualifier is used to indicate to the iSER layer 
whether or not it should generate an eventual local completion 
notification to the iSCSI layer. See Section 3.2.2 on 

Data Completion Notify for details. 


3. Get Data 


Input qualifiers: Connection Handle, content of an R2T PDU, 
Data Descriptor, Notify Enable 


Return results: Not specified 


This is used by the iSCSI layer at the target to request the inbound 
transfer of solicited data requested by an R2T PDU into the buffer 
identified by the Data Descriptor qualifier. The iSCSI layer can 
only invoke the Get Data Operational Primitive when the connection is 
in iSER-assisted mode. 


The Notify Enable qualifier is used to indicate to the iSER layer 
whether or not it should generate the eventual local completion 
notification to the iSCSI layer. See Section 3.2.2 on 

Data Completion Notify for details. 


4. Allocate Connection Resources 


Input qualifiers: Connection Handle, Resource Descriptor 
(optional) 


Return results: Status 


This is used by the iSCSI layers at the initiator and the target to 
request the allocation of all connection resources necessary to 
support RCaP for an operational iSCSI/iSER connection. The iSCSI 
layer may optionally specify the implementation-specific resource 
requirements for the iSCSI connection using the Resource Descriptor 
qualifier. 


A return result of Status-success means that the invocation 


succeeded, and a return result of Status-failure means that the 
invocation failed. If the invocation is for a Connection Handle for 
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which an earlier invocation succeeded, the request will be ignored by 
the iSER layer and the result of Status=success will be returned. 
Only one Allocate_Connection_Resources Operational Primitive 
invocation can be outstanding for a given Connection_Handle at any 
time. 


3.1.5. Deallocate_Connection_Resources 
Input qualifiers: Connection Handle 
Return results: Not specified 
This is used by the iSCSI layers at the initiator and the target to 
request the deallocation of all connection resources that were 


allocated earlier as a result of a successful invocation of the 
Allocate Connection Resources Operational Primitive. 


3.1.6. Enable Datamover 


Input qualifiers: Connection Handle, 
Transport Connection Descriptor, Final Login Response PDU 
(optional) 


Return results: Not specified 


This is used by the iSCSI layers at the initiator and the target to 
request that a specified iSCSI connection be transitioned to iSER- 
assisted mode. The Transport Connection Descriptor qualifier is used 
to identify the specific connection associated with the 

Connection Handle. The iSCSI layer can only invoke the 

Enable Datamover Operational Primitive when there is a corresponding 
prior resource allocation. 


The Final Login Response PDU input qualifier is applicable only for a 
target, and contains the final Login Response PDU that concludes the 
iSCSI Login Phase. If the underlying transport is TCP, the final 
Login Response PDU must be sent as a byte stream as expected by the 
iSCSI layer at the initiator. When this qualifier is used, the iSER 
layer at the target MUST transmit this final Login Response PDU 
before transitioning to iSER-assisted mode. 


3.1.7. Connection Terminate 
Input qualifiers: Connection Handle 


Return results: Not specified 
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This is used by the iSCSI layers at the initiator and the target to 
request that a specified iSCSI/iSER connection be terminated and all 
associated connection and task resources be freed. When this 
Operational Primitive invocation returns to the iSCSI layer, the 
iSCSI layer may assume full ownership of all iSCSI-level resources, 
e.g., I/O Buffers, associated with the connection. 


3.1.8. Notice_ Key Values 


Input qualifiers: Connection Handle, number of keys, list of 
Key-Value pairs 


Return results: Not specified 


This is used by the iSCSI layers at the initiator and the target to 
request that the iSER layer take note of the specified Key-Value 
pairs that were negotiated by the iSCSI peers for the connection. 


3.1.9. Deallocate Task Resources 
Input qualifiers: Connection Handle, ITT 
Return results: Not specified 


This is used by the iSCSI layers at the initiator and the target to 
request the deallocation of all RCaP-specific resources allocated by 
the iSER layer for the task identified by the ITT qualifier. The 
iSER layer may require a certain number of RCaP-specific resources 
associated with the ITT for each new iSCSI task. In the normal 
course of execution, these task-level resources in the iSER layer are 
assumed to be transparently allocated on each task initiation and 
deallocated on the conclusion of each task as appropriate. In 
exception scenarios where the task does not conclude with a SCSI 
Response PDU, the iSER layer needs to be notified of the individual 
task terminations to aid its task-level resource management. This 
Operational Primitive is used for this purpose, and is not needed 
when a SCSI Response PDU normally concludes a task. Note that RCaP- 
Specific task resources are deallocated by the iSER layer when a SCSI 
Response PDU normally concludes a task, even if the SCSI status was 
not success. 


3.2. Operational Primitives Used by iSER 
The iSER layer MUST use the following Operational Primitives offered 


by the iSCSI protocol layer when the connection is in iSER-assisted 
mode. 
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.1. Control Notify 
Input qualifiers: Connection Handle, an iSCSI control-type PDU 
Return results: Not specified 


This is used by the iSER layers at the initiator and the target to 
notify the iSCSI layer of the availability of an inbound iSCSI 
control-type PDU. A PDU is described as "available" to the iSCSI 
layer when the iSER layer notifies the iSCSI layer of the reception 
of that inbound PDU, along with an implementation-specific indication 
as to where the received PDU is. 


.2. Data Completion Notify 
Input qualifiers: Connection Handle, ITT, SN 
Return results: Not specified 


This is used by the iSER layer to notify the iSCSI layer of the 
completion of outbound data transfer that was requested by the iSCSI 
layer only if the invocation of the Put Data Operational Primitive 
(see Section 3.1.2) was qualified with Notify Enable set. SN refers 
to the DataSN associated with the SCSI Data-in PDU. 


This is used by the iSER layer to notify the iSCSI layer of the 
completion of inbound data transfer that was requested by the iSCSI 
layer only if the invocation of the Get Data Operational Primitive 
(see Section 3.1.3) was qualified with Notify Enable set. SN refers 
to the R2TSN associated with the R2T PDU. 


3. Data ACK Notify 
Input qualifier: Connection Handle, ITT, DataSN 
Return results: Not specified 


This is used by the iSER layer at the target to notify the iSCSI 
layer of the arrival of the data acknowledgement (as defined in 
[RFC3720]) requested earlier by the iSCSI layer for the outbound data 
transfer via an invocation of the Put Data Operational Primitive 
where the A-bit in the SCSI Data-in PDU is set to 1. See Section 
7.3.5. DataSN refers to the expected DataSN of the next SCSI Data-in 
PDU, which immediately follows the SCSI Data-in PDU with the A-bit 
set to which this notification corresponds, with semantics as defined 
in [RFC3720]. 
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.4. Connection Terminate Notify 
Input qualifiers: Connection Handle 
Return results: Not specified 


This is used by the iSER layers at the initiator and the target to 
notify the iSCSI layer of the unsolicited termination or failure of 
an iSCSI/iSER connection. The iSER layer MUST deallocate the 
connection and task resources associated with the terminated 
connection before the invocation of this Operational Primitive. Note 
that the Connection Terminate Notify Operational Primitive is not 
invoked when the termination of the connection is earlier requested 
by the local iSCSI layer. 


iSCSI Protocol Usage Requirements 


To operate in an iSER-assisted mode, the iSCSI layers at both the 
initiator and the target MUST negotiate the RDMAExtensions key (see 
Section 6.3) to "Yes" on the leading connection. If the 
RDMAExtensions key is not negotiated to "Yes", then iSER-assisted 
mode MUST NOT be used. If the RDMAExtensions key is negotiated to 
"Yes" but the invocation of the Allocate Connection Resources 
Operational Primitive to the iSER layer fails, the iSCSI layer MUST 
fail the iSCSI Login process or terminate the connection as 
appropriate. See Section 10.1.3.1 for details. 


If the RDMAExtensions key is negotiated to "Yes", the iSCSI layer 
MUST satisfy the following protocol usage requirements from the iSER 
protocol: 


1. The iSCSI layer at the initiator MUST set ExpDataSN to 0 in Task 
Management Function Requests for Task Allegiance Reassignment for 
read/bidirectional commands, so as to cause the target to send 
all unacknowledged read data. 


2. The iSCSI layer at the target MUST always return the SCSI status 
in a separate SCSI Response PDU for read commands, i.e., there 
MUST NOT be a "phase collapse" in concluding a SCSI read command. 


3. The iSCSI layers at both the initiator and the target MUST 
support the keys as defined in Section 6 on Login/Text 
Operational Keys. If used as specified, these keys MUST NOT be 
answered with NotUnderstood, and the semantics as defined MUST be 
followed for each iSER-assisted connection. 


4. The iSCSI layer at the initiator MUST NOT issue SNACKs for PDUs. 
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Lower Layer Interface Requirements 


Interactions with the RCaP Layer 


The iSER protocol layer is layered on top of an RCaP layer (see 
Figure 1) and the following are the key features that are assumed to 
be supported by any RCaP layer: 


* 


The RCaP layer supports all basic RDMA operations, including RDMA 
Write Operation, RDMA Read Operation, Send Operation, Send with 
Invalidate Operation, Send with Solicited Event Operation, Send 
with Solicited Event and Invalidate Operation, and Terminate 
Operation. 


The RCaP layer provides reliable, in-order message delivery and 
direct data placement. 


When the iSER layer initiates an RDMA Read Operation following an 
RDMA Write Operation on one RCaP Stream, the RDMA Read Response 
Message processing on the remote node will be started only after 
the preceding RDMA Write Message payload is placed in the memory 
of the remote node. 


The RCaP layer encapsulates a single iSER Message into a single 
RCaP Message on the Data Source side. The RCaP layer decapsulates 
the iSER Message before delivering it to the iSER layer on the 
Data Sink side. 


When the iSER layer provides the STag to be remotely invalidated 
to the RCaP layer for a SendInvSE Message, the RCaP layer uses 
this STag as the STag to be invalidated in the SendInvSE Message. 


The RCaP layer uses the STag and Tagged Offset provided by the 
iSER layer for the RDMA Write and RDMA Read Request Messages. 


When the RCaP layer delivers the content of an RDMA Send Message 
Type to the iSER layer, the RCaP layer provides the length of the 
RDMA Send message. This ensures that the iSER layer does not have 
to carry a length field in the iSER header. 


When the RCaP layer delivers the SendSE or SendInvSE Message to 
the iSER layer, it notifies the iSER layer with the mechanism 
provided on that interface. 


When the RCaP layer delivers a SendInvSE Message to the iSER 
layer, it passes the value of the STag that was invalidated. 
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* The RCaP layer propagates all status and error indications to the 
iSER layer. 


* For a transport layer that operates in byte stream mode such as 
TCP, the RCaP implementation supports the enabling of the RDMA 
mode after connection establishment and the exchange of Login 
parameters in byte stream mode. For a transport layer that 
provides message delivery capability such as [IB], the RCaP 
implementation supports the use of the messaging capability by the 
iSCSI layer directly for the Login Phase after connection 
establishment before enabling iSER-assisted mode. 


* Whenever the iSER layer terminates the RCaP Stream, the RCaP layer 
terminates the associated connection. 


Interactions with the Transport Layer 


The iSER layer does not directly setup the transport layer connection 
(e.g., TCP, or [IB]). During connection setup, the iSCSI layer is 
responsible for setting up the connection. If the login is 
successful, the iSCSI layer invokes the Enable Datamover Operational 
Primitive to request the iSER layer to transition to the iSER- 
assisted mode for that iSCSI connection. See Section 5.1 on 
iSCSI/iSER connection setup. After transitioning to iSER-assisted 
mode, the RCaP layer and the underlying transport layer are 
responsible for maintaining the connection and reporting to the iSER 
layer any connection failures. 


Connection Setup and Termination 
iSCSI/iSER Connection Setup 


During connection setup, the iSCSI layer at the initiator is 
responsible for establishing a connection with the target. After the 
connection is established, the iSCSI layers at the initiator and the 
target enter the Login Phase using the same rules as outlined in 
[RFC3720]. Transition to iSER-assisted mode occurs when the 
connection transitions into the iSCSI Full Feature Phase following a 
successful login negotiation between the initiator and the target in 
which iSER-assisted mode is negotiated and the connection resources 
necessary to support RCaP have been allocated at both the initiator 
and the target. The same connection MUST be used for both the iSCSI 
Login Phase and the subsequent iSER-assisted Full Feature Phase. 


iSER-assisted mode MUST be enabled only if it is negotiated on the 
leading connection during the LoginOperationalNegotiation stage of 
the iSCSI Login Phase.  iSER-assisted mode is negotiated using the 
RDMAExtensions-«boolean-value» key. Both the initiator and the 


et al. Standards Track [Page 27] 


RFC 5046 iSER Specification October 2007 


target MUST exchange the RDMAExtensions key with the value set to 
"Yes" to enable iSER-assisted mode. If both the initiator and the 
target fail to negotiate the RDMAExtensions key set to "Yes", then 
the connection MUST continue with the login semantics as defined in 


[RFC3720]. If the RDMAExtensions key is not negotiated to Yes, then 
for some RCaP implementation (such as [IB]), the connection may need 
to be re-established in TCP capable mode. (For InfiniBand this will 


require an [IPOIB] type connection.) 


iSER-assisted mode is defined for a Normal session only and the 
RDMAExtensions key MUST NOT be negotiated for a Discovery session. 
Discovery sessions are always conducted using the transport layer as 
described in [RFC3720]. 


An iSER enabled node is not required to initiate the RDMAExtensions 
key exchange if its preference is for the Traditional iSCSI mode. 
The RDMAExtensions key, if offered, MUST be sent in the first 
available Login Response or Login Request PDU in the 
LoginOperationalNegotiation stage. This is due to the fact that the 
value of some login parameters might depend on whether iSER-assisted 
mode is enabled. 


iSER-assisted mode is a session-wide attribute. If both the 
initiator and the target negotiate RDMAExtensions-"Yes" on the 
leading connection of a session, then all subsequent connections of 
the same session MUST enable iSER-assisted mode without having to 
exchange an RDMAExtensions key during the iSCSI Login Phase. 


Conversely, if both the initiator and the target fail to negotiate 
RDMAExtensions to "Yes" on the leading connection of a session, then 
the RDMAExtensions key MUST NOT be negotiated further on any 
additional subsequent connection of the session. 


When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest 
and the DataDigest keys MUST be negotiated to "None" on all 
iSCSI/iSER connections participating in that iSCSI session. This is 
because, for an iSCSI/iSER connection, RCaP is responsible for 
providing error detection that is at least as good as a 32-bit CRC 
for all iSER Messages. Furthermore, all SCSI Read data are sent 
using RDMA Write Messages instead of the SCSI Data-in PDUs, and all 
solicited SCSI write data are sent using RDMA Read Response Messages 
instead of the SCSI Data-out PDUs.  HeaderDigest and DataDigest that 
apply to iSCSI PDUs, would not be appropriate for RDMA Read and RDMA 
Write operations used with iSER. 
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Ta Initiator Behavior 


If the outcome of the iSCSI negotiation is to enable iSER-assisted 
mode, then on the initiator side, prior to sending the Login Request 
with the T (Transit) bit set to 1 and the NSG (Next Stage) field set 
to FullFeaturePhase, the iSCSI layer MUST request that the iSER layer 
allocate the connection resources necessary to support RCaP by 
invoking the Allocate_Connection_Resources Operational Primitive. 
The connection resources required are defined by implementation and 
are outside the scope of this specification. The iSCSI layer may 
invoke the Notice_Key_Values Operational Primitive before invoking 
the Allocate_Connection_Resources Operational Primitive to request 
that the iSER layer take note of the negotiated values of the iSCSI 
keys for the connection. The specific keys to be passed as input 
qualifiers are implementation dependent. These may include, but are 
not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc. 


To minimize the potential for a denial-of service attack, the iSCSI 
layer MUST NOT request that the iSER layer allocate the connection 
resources necessary to support RCaP until the iSCSI layer is 
sufficiently far along in the iSCSI Login Phase that it is reasonably 
certain that the peer side is not an attacker. In particular, if the 
Login Phase includes a SecurityNegotiation stage, the iSCSI layer 
MUST defer the connection resource allocation (i.e., invoking the 
Allocate_Connection_Resources Operational Primitive) to the 
LoginOperationalNegotiation stage [RFC3720] so that the resource 
allocation occurs after the authentication phase is completed. 


Among the connection resources allocated at the initiator is the 
Inbound RDMA Read Queue Depth (IRD). As described in Section 9.5.1, 
R2Ts are transformed by the target into RDMA Read operations. IRD 
limits the maximum number of simultaneously incoming outstanding RDMA 
Read Requests per an RCaP Stream from the target to the initiator. 
The required value of IRD is outside the scope of the iSER 
specification. The iSER layer at the initiator MUST set IRD to 1 or 
higher if R2Ts are to be used in the connection. However, the iSER 
layer at the initiator MAY set IRD to 0 based on implementation 
configuration, which indicates that no R2Ts will be used on that 
connection. Initially, the iSER-IRD value at the initiator SHOULD be 
set to the IRD value at the initiator and MUST NOT be more than the 
IRD value. 


On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be 


set to 0, since the iSER layer at the initiator does not issue RDMA 
Read Requests to the target. 
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Failure to allocate the requested connection resources locally 
results in a login failure and its handling is described in Section 
10.1.31. 


If the iSER layer at the initiator is successful in allocating the 
connection resources necessary to support RCaP, the following events 
MUST occur in the specified sequence: 


1. The iSER layer MUST return a success status to the iSCSI layer in 
response to the Allocate Connection Resources Operational 
Primitive. 


2. After the target returns the Login Response with the T bit set to 
1 and the NSG field set to FullFeaturePhase, and a status class 
of 0 (Success), the iSCSI layer MUST request that the iSER layer 
transition to iSER-assisted mode by invoking the Enable Datamover 


Operational Primitive with the following qualifiers. (See 
Section 10.1.4.6 for the case when the status class is not 
Success.): 


a. Connection Handle that identifies the iSCSI connection. 


b. Transport Connection Descriptor that identifies the specific 
transport connection associated with the Connection Handle. 


3. If necessary, the iSER layer should enable RCaP and transition 
the connection to iSER-assisted mode. When the RCaP is iWARP, 
then this step MUST be done. Not all RCaPs may need it depending 
on the RCaP Stream start-up state. 


4. The iSER layer MUST send the iSER Hello Message as the first iSER 
Message. See Section 5.1.3 on iSER Hello Exchange. 


.2. Target Behavior 


If the outcome of the iSCSI negotiation is to enable iSER-assisted 
mode, then on the target side, prior to sending the Login Response 
with the T (Transit) bit set to 1 and the NSG (Next Stage) field set 
to FullFeaturePhase, the iSCSI layer MUST request that the iSER layer 
allocate the resources necessary to support RCaP by invoking the 
Allocate Connection Resources Operational Primitive. The connection 
resources required are defined by implementation and are outside the 
Scope of this specification.  Optionally, the iSCSI layer may invoke 
the Notice Key Values Operational Primitive before invoking the 
Allocate Connection Resources Operational Primitive to request that 
the iSER layer take note of the negotiated values of the iSCSI keys 
for the connection. The specific keys to be passed as input 
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qualifiers are implementation dependent. These may include, but are 
not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc. 


To minimize the potential for a denial-of-service attack, the iSCSI 
layer MUST NOT request that the iSER layer allocate the connection 
resources necessary to support RCaP until the iSCSI layer is 
sufficiently far along in the iSCSI Login Phase that it is reasonably 
certain that the peer side is not an attacker. In particular, if the 
Login Phase includes a SecurityNegotiation stage, the iSCSI layer 
MUST defer the connection resource allocation (i.e., invoking the 
Allocate Connection Resources Operational Primitive) to the 
LoginOperationalNegotiation stage [RFC3720] so that the resource 
allocation occurs after the authentication phase is completed. 


Among the connection resources allocated at the target is the 
Outbound RDMA Read Queue Depth (ORD). As described in Section 9.5.1, 
R2Ts are transformed by the target into RDMA Read operations. The 
ORD limits the maximum number of simultaneously outstanding RDMA Read 
Requests per RCaP Stream from the target to the initiator. 

Initially, the iSER-ORD value at the target SHOULD be set to the ORD 
value at the target. 


On the other hand, the IRD at the target MAY be set to 0 since the 
iSER layer at the target does not expect RDMA Read Requests to be 
issued by the initiator. 


Failure to allocate the requested connection resources locally 
results in a login failure and its handling is described in Section 
10.1381: 


If the iSER layer at the target is successful in allocating the 
connection resources necessary to support RCaP, the following events 
MUST occur in the specified sequence: 


1. The iSER layer MUST return a success status to the iSCSI layer in 
response to the Allocate_Connection_Resources Operational 
Primitive. 

2. The iSCSI layer MUST request that the iSER layer transition to 
iSER-assisted mode by invoking the Enable_Datamover Operational 
Primitive with the following qualifiers: 


a. Connection_Handle that identifies the iSCSI connection. 


b. Transport Connection Descriptor that identifies the specific 
transport connection associated with the Connection Handle. 
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C. The final transport layer (e.g., TCP) message containing the 
Login Response with the T bit set to 1 and the NSG field set 
to FullFeaturePhase. 


3. The iSER layer MUST send the final Login Response PDU in the 
native transport mode to conclude the iSCSI Login Phase. If the 
underlying transport is TCP, then the iSER layer MUST send the 
final Login Response PDU in byte stream mode. 


4. After sending the final Login Response PDU, the iSER layer should 
enable RCaP if necessary and transition the connection to iSER- 
assisted mode. When the RCaP is iWARP, then this step MUST be 
done. Not all RCaPs may need it depending on the RCaP Stream 
start-up state. 


5. After receiving the iSER Hello Message from the initiator, the 
iSER layer MUST respond with the iSER HelloReply Message to be 
sent as the first iSER Message. See Section 5.1.3 on iSER Hello 
Exchange for more details. 


Note: In the above sequence, the operations as described in bullets 3 
and 4 MUST be performed atomically for iWARP connections. Failure to 
do this may result in race conditions. 


3. iSER Hello Exchange 


After the connection transitions into iSER-assisted mode, the first 
iSER Message sent by the iSER layer at the initiator to the target 
MUST be the iSER Hello Message. The iSER Hello Message is used by 
the iSER layer at the initiator to declare iSER parameters to the 
target. See Section 9.3 on iSER Header Format for the iSER Hello 
Message. 


In response to the iSER Hello Message, the iSER layer at the target 
MUST return the iSER HelloReply Message as the first iSER Message 
sent by the target. The iSER HelloReply Message is used by the iSER 
layer at the target to declare iSER parameters to the initiator. See 
Section 9.4 on iSER Header Format for the iSER HelloReply Message. 


In the iSER Hello Message, the iSER layer at the initiator declares 
the iSER-IRD value to the target. 


Upon receiving the iSER Hello Message, the iSER layer at the target 
MUST set the iSER-ORD value to the minimum of the iSER-ORD value at 
the target and the iSER-IRD value declared by the initiator. The 
iSER layer at the target MAY adjust (lower) its ORD value to match 
the iSER-ORD value if the iSER-ORD value is smaller than the ORD 
value at the target in order to free up the unused resources. 
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In the iSER HelloReply Message, the iSER layer at the target declares 
the iSER-ORD value to the initiator. 


Upon receiving the iSER HelloReply Message, the iSER layer at the 
initiator MAY adjust (lower) its IRD value to match the iSER-ORD 
value in order to free up the unused resources, if the iSER-ORD value 
declared by the target is smaller than the iSER-IRD value declared by 
the initiator. 


It is an iSER level negotiation failure if the iSER parameters 
declared in the iSER Hello Message by the initiator are unacceptable 
to the target. This includes the following: 


* The initiator-declared iSER-IRD value is greater than 0 and the 
target-declared iSER-ORD value is 0. 


* The initiator-supported and the target-supported iSER protocol 
versions do not overlap. 


See Section 10.1.3.2 for the handling of the error situation. 
iSCSI/iSER Connection Termination 
1. Normal Connection Termination at the Initiator 


The iSCSI layer at the initiator terminates an iSCSI/iSER connection 
normally by invoking the Send Control Operational Primitive qualified 
with the Logout Request PDU. The iSER layer at the initiator MUST 
use a SendSE Message to send the Logout Request PDU to the target. 
After the iSER layer at the initiator receives the SendSE Message 
containing the Logout Response PDU from the target, it MUST notify 
the iSCSI layer by invoking the Control Notify Operational Primitive 
qualified with the Logout Response PDU. 


After the iSCSI logout process is complete, the iSCSI layer at the 
target is responsible for closing the iSCSI/iSER connection as 
described in Section 5.2.2. After the RCaP layer at the initiator 
reports that the connection has been closed, the iSER layer at the 
initiator MUST deallocate all connection and task resources (if any) 
associated with the connection, and invalidate the Local Mapping (s) 
(if any) that associate the ITT(s) used on that connection to the 
local STag(s) before notifying the iSCSI layer by invoking the 
Connection Terminate Notify Operational Primitive. 
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-2. Normal Connection Termination at the Target 


Upon receiving the SendSE Message containing the Logout Request PDU, 
the iSER layer at the target MUST notify the iSCSI layer at the 
target by invoking the Control_Notify Operational Primitive qualified 
with the Logout Request PDU. The iSCSI layer completes the logout 
process by invoking the Send_Control Operational Primitive qualified 
with the Logout Response PDU. The iSER layer at the target MUST use 
a SendSE Message to send the Logout Response PDU to the initiator. 
After the iSCSI logout process is complete, the iSCSI layer at the 
target MUST request that the iSER layer at the target terminate the 
RCaP Stream by invoking the Connection_Terminate Operational 
Primitive. 


As part of the termination process, the RCaP layer MUST close the 
connection. When the RCaP layer notifies the iSER layer after the 
RCaP Stream and the associated connection are terminated, the iSER 
layer MUST deallocate all connection and task resources (if any) 
associated with the connection, and invalidate the Local and Remote 
Mapping(s) (if any) that associate the ITT(s) used on that connection 
to the local STag(s) and the Advertised STag(s) respectively. 


.3. Termination without Logout Request/Response PDUs 


5.2.3.1. Connection Termination Initiated by the iSCSI Layer 


Ko, 


The Connection_Terminate Operational Primitive MAY be invoked by the 
iSCSI layer to request that the iSER layer terminate the RCaP Stream 
without having previously exchanged the Logout Request and Logout 
Response PDUs between the two iSCSI/iSER nodes. As part of the 
termination process, the RCaP layer will close the connection. When 
the RCaP layer notifies the iSER layer after the RCaP Stream and the 
associated connection are terminated, the iSER layer MUST perform the 
following actions. 


If the Connection_Terminate Operational Primitive is invoked by the 
iSCSI layer at the target, then the iSER layer at the target MUST 
deallocate all connection and task resources (if any) associated with 
the connection, and invalidate the Local and Remote Mappings (if any) 
that associate the ITT(s) used on the connection to the local STag(s) 
and the Advertised STag(s), respectively. 


If the Connection_Terminate Operational Primitive is invoked by the 
iSCSI layer at the initiator, then the iSER layer at the initiator 
MUST deallocate all connection and task resources (if any) associated 
with the connection, and invalidate the Local Mapping(s) (if any) 
that associate the ITT(s) used on the connection to the local 
STag(s). 
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If the iSCSI/iSER connection is terminated without the invocation of 
Connection_Terminate from the iSCSI layer, the iSER layer MUST notify 
the iSCSI layer that the iSCSI/iSER connection has been terminated by 
invoking the Connection_Terminate_Notify Operational Primitive. 


Prior to invoking Connection_Terminate_Notify, the iSER layer at the 
target MUST deallocate all connection and task resources (if any) 
associated with the connection, and invalidate the Local and Remote 
Mappings (if any) that associate the ITT(s) used on the connection to 
the local STag(s) and the Advertised STag(s), respectively. 


Prior to invoking Connection_Terminate_Notify, the iSER layer at the 
initiator MUST deallocate all connection and task resources (if any) 
associated with the connection, and invalidate the Local Mappings (if 
any) that associate the ITT(s) used on the connection to the local 
STag(s). 


If the remote iSCSI/iSER node initiated the closing of the connection 
(e.g., by sending a TCP FIN or TCP RST), the iSER layer MUST notify 
the iSCSI layer after the RCaP layer reports that the connection is 
closed by invoking the Connection_Terminate_Notify Operational 
Primitive. 


Another example of a connection termination without a preceding 
logout is when the iSCSI layer at the initiator does an implicit 
logout (connection reinstatement). 


6. Login/Text Operational Keys 


Certain iSCSI login/text operational keys have restricted usage in 
iSER, and additional keys are used to support the iSER protocol 
functionality. All other keys defined in [RFC3720] and not discussed 
in this section may be used on iSCSI/iSER connections with the same 
semantics. 


6.1. HeaderDigest and DataDigest 
Irrelevant when: RDMAExtensions=Yes 
Negotiations resulting in RDMAExtensions=Yes for a session implies 


HeaderDigest=None and DataDigest=None for all connections in that 
session and overrides both the default and an explicit setting. 
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MaxRecvDataSegmentLength 


For an iSCSI connection belonging to a session in which 
RDMAExtensions=Yes was negotiated on the leading connection of the 
session, MaxRecvDataSegmentLength need not be declared in the Login 
Phase. Instead, InitiatorRecvDataSegmentLength (as described in 
Section 6.5) and TargetRecvDataSegmentLength (as described in Section 
6.4) keys are negotiated. The values of the local and remote 
MaxRecvDataSegmentLength are derived from the 
InitiatorRecvDataSegmentLength and TargetRecvDataSegmentLength keys 
even if the MaxRecvDataSegmentLength is declared during the Login 
Phase. 


In the Full Feature Phase, the initiator MUST consider the value of 
its local MaxRecvDataSegmentLength (that it would have declared to 
the target) as having the value of InitiatorRecvDataSegmentLength, 
and the value of the remote MaxRecvDataSegmentLength (that would have 
been declared by the target) as having the value of 
TargetRecvDataSegmentLength. Similarly, the target MUST consider the 
value of its local MaxRecvDataSegmentLength (that it would have 
declared to the initiator) as having the value of 
TargetRecvDataSegmentLength, and the value of the remote 
MaxRecvDataSegmentLength (that would have been declared by the 
initiator) as having the value of InitiatorRecvDataSegmentLength. 


The MaxRecvDataSegmentLength key is applicable only for iSCSI 
control-type PDUs. 


RDMAExtensions 

Use: LO (leading only) 

Senders: Initiator and Target 

Scope: SW (session-wide) 

RDMAExtensions-«boolean-value» 

Irrelevant when: SessionType-Discovery 

Default is No 

Result function is AND 

This key is used by the initiator and the target to negotiate support 


for iSER-assisted mode. To enable the use of iSER-assisted mode, 
both the initiator and the target MUST exchange RDMAExtensions-Yes. 
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iSER-assisted mode MUST NOT be used if either the initiator or the 
target offers RDMAExtensions=No. 


An iSER-enabled node is not required to initiate the RDMAExtensions 
key exchange if it prefers to operate in the Traditional iSCSI mode. 
However, if the RDMAExtensions key is to be negotiated, an initiator 
MUST offer the key in the first Login Request PDU in the 
LoginOperationalNegotiation stage of the leading connection, anda 
target MUST offer the key in the first Login Response PDU with which 
it is allowed to do so (i.e., the first Login Response PDU issued 
after the first Login Request PDU with the C bit set to 0) in the 
LoginOperationalNegotiation stage of the leading connection. In 
response to the offered key=value pair of RDMAExtensions=yes, an 
initiator MUST respond in the next Login Request PDU with which it is 
allowed to do so, and a target MUST respond in the next Login 
Response PDU with which it is allowed to do so. 


Negotiating the RDMAExtensions key first enables a node to negotiate 
the optimal value for other keys. Certain iSCSI keys such as 
MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T, 
ImmediateData, etc., may be negotiated differently depending on 
whether the connection is in Traditional iSCSI mode or iSER-assisted 
mode. 

TargetRecvDataSegmentLength 

Use: IO (Initialize only) 

Senders: Initiator and Target 

Scope: CO (connection-only) 


Irrelevant when: RDMAExtensions=No 


TargetRecvDataSegmentLength-«numerical-value-512-to- (2**24-1)» 


Default is 8192 bytes 
Result function is minimum 


This key is relevant only for the iSCSI connection of an iSCSI 
session if RDMAExtensions-Yes is negotiated on the leading connection 
of the session. It is used by the initiator and target to negotiate 
the maximum size of the data segment that an initiator may send to 
the target in an iSCSI control-type PDU in the Full Feature Phase. 
For SCSI Command PDUs and SCSI Data-out PDUs containing non-immediate 
unsolicited data to be sent by the initiator, the initiator MUST send 
all non-Final PDUs with a data segment size of exactly 


et al. Standards Track [Page 37] 


RFC 5046 iSER Specification October 2007 


TargetRecvDataSegmentLength whenever the PDUs constitute a data 
sequence whose size is larger than TargetRecvDataSegmentLength. 


6.5. InitiatorRecvDataSegmentLength 
Use: IO (Initialize only) 
Senders: Initiator and Target 
Scope: CO (connection-only) 
Irrelevant when: RDMAExtensions=No 


InitiatorRecvDataSegmentLength-«numerical-value-512-to- (2**24-1)» 


Default is 8192 bytes 
Result function is minimum 
This key is relevant only for the iSCSI connection of an iSCSI 
session if RDMAExtensions-Yes is negotiated on the leading connection 
of the session. It is used by the initiator and target to negotiate 
the maximum size of the data segment that a target may send to the 
initiator in an iSCSI control-type PDU in the Full Feature Phase. 
6.6. OFMarker and IFMarker 
Irrelevant when: RDMAExtensions-Yes 
Negotiations resulting in RDMAExtensions-Yes for a session implies 
OFMarker=No and IFMarker=No for all connections in that session and 
overrides both the default and an explicit setting. 
6.7. MaxOutstandingUnexpectedPDUs 
Use: LO (leading only), Declarative 
Senders: Initiator and Target 


Scope: SW (session-wide) 


Irrelevant when: RDMAExtensions-No 


MaxOutstandingUnexpectedPDUs-«numerical-value-from-2-to- (2**32-1) | 
0» 


Default is 0 
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This key is used by the initiator and the target to declare the 
maximum number of outstanding "unexpected" iSCSI control-type PDUs 
that it can receive in the Full Feature Phase. It is intended to 
allow the receiving side to determine the amount of buffer resources 
needed beyond the normal flow control mechanism available in iSCSI. 
An initiator or target should select a value such that it would not 
impose an unnecessary constraint on the iSCSI layer under normal 
circumstances. The value of 0 is defined to indicate that the 
declarer has no limit on the maximum number of outstanding 
"unexpected" iSCSI control-type PDUs that it can receive. See 
Sections 8.1.1 and 8.1.2 for the usage of this key. Note that iSER 
Hello and HelloReply Messages are not iSCSI control-type PDUs and are 
not affected by this key. 


iSCSI PDU Considerations 


When a connection is in the iSER-assisted mode, two types of message 
transfers are allowed between the iSCSI layer at the initiator and 
the iSCSI layer at the target. These are known as the iSCSI data- 
type PDUs and the iSCSI control-type PDUs, and these terms are 
described in the following sections. 


iSCSI Data-Type PDU 


An iSCSI data-type PDU is defined as an iSCSI PDU that causes data 
transfer, transparent to the remote iSCSI layer, to take place 
between the peer iSCSI nodes in the full feature phase of an 
iSCSI/iSER connection. An iSCSI data-type PDU, when requested for 
transmission by the iSCSI layer in the sending node, results in the 
data being transferred without the participation of the iSCSI layers 
at the sending and the receiving nodes. This is due to the fact that 
the PDU itself is not delivered as-is to the iSCSI layer in the 
receiving node. Instead, the data transfer operations are 
transformed into the appropriate RDMA operations that are handled by 
the RDMA-Capable Controller. The set of iSCSI data-type PDUs 
consists of SCSI Data-in PDUs and R2T PDUs. 


If the invocation of the Operational Primitive by the iSCSI layer to 
request that the iSER layer process an iSCSI data-type PDU is 
qualified with Notify_Enable set, then upon completing the RDMA 
operation, the iSER layer at the target MUST notify the iSCSI layer 
at the target by invoking the Data_Completion_Notify Operational 
Primitive qualified with ITT and SN. There is no data completion 
notification at the initiator since the RDMA operations are 
completely handled by the RDMA-Capable Controller at the initiator 
and the iSER layer at the initiator is not involved with the data 
transfer associated with iSCSI data-type PDUs. 
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If the invocation of the Operational Primitive by the iSCSI layer to 
request that the iSER layer process an iSCSI data-type PDU is 
qualified with Notify_Enable cleared, then upon completing the RDMA 
operation, the iSER layer at the target MUST NOT notify the iSCSI 
layer at the target and MUST NOT invoke the Data_Completion_Notify 
Operational Primitive. 


If an operation associated with an iSCSI data-type PDU fails for any 
reason, the contents of the Data Sink buffers associated with the 
operation are considered indeterminate. 


iSCSI Control-Type PDU 


Any iSCSI PDU that is not an iSCSI data-type PDU and also not a SCSI 
Data-out PDU carrying solicited data is defined as an iSCSI control- 
type PDU. The iSCSI layer invokes the Send_Control Operational 
Primitive to request that the iSER layer process an iSCSI control- 
type PDU. iSCSI control-type PDUs are transferred using Send Message 
Types of RCaP. Specifically, note that SCSI Data-out PDUs carrying 
unsolicited data are defined as iSCSI control-type PDUs. See Section 
7.3.4 on the treatment of SCSI Data-out PDUs. 


When the iSER layer receives an iSCSI control-type PDU, it MUST 
notify the iSCSI layer by invoking the Control_Notify Operational 
Primitive qualified with the iSCSI control-type PDU. 


iSCSI PDUs 


This section describes the handling of each of the iSCSI PDU types by 
the iSER layer. The iSCSI layer requests that the iSER layer process 
the iSCSI PDU by invoking the appropriate Operational Primitive. A 
Connection_Handle MUST qualify each of these invocations. In 
addition, BHS and the optional AHS of the iSCSI PDU as defined in 
[RFC3720] MUST qualify each of the invocations. The qualifying 
Connection_Handle, the BHS, and the AHS are not explicitly listed in 
the subsequent sections. 


1. SCSI Command 
Type: control-type PDU 


PDU-specific qualifiers (for SCSI Write or bidirectional command) : 
ImmediateDataSize, UnsolicitedDataSize, DataDescriptorOut 


PDU-specific qualifiers (for SCSI read or bidirectional command) : 
DataDescriptorIn 
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The iSER layer at the initiator MUST send the SCSI command in a 
SendSE Message to the target. 


For a SCSI Write or bidirectional command, the iSCSI layer at the 
initiator MUST invoke the Send_Control Operational Primitive as 
follows: 


* If there is immediate data to be transferred for the SCSI Write or 
bidirectional command, the qualifier ImmediateDataSize MUST be 
used to define the number of bytes of immediate unsolicited data 
to be sent with the Write or bidirectional command, and the 
qualifier DataDescriptorOut MUST be used to define the initiator’s 
I/O Buffer containing the SCSI Write data. 


* If there is unsolicited data to be transferred for the SCSI Write 
or bidirectional command, the qualifier UnsolicitedDataSize MUST 
be used to define the number of bytes of immediate and non- 
immediate unsolicited data for the command. The iSCSI layer will 
issue one or more SCSI Data-out PDUs for the non-immediate 
unsolicited data. See Section 7.3.4 on SCSI Data-out. 


* If there is solicited data to be transferred for the SCSI write or 
bidirectional command, as indicated by the Expected Data Transfer 
Length in the SCSI Command PDU exceeding the value of 
UnsolicitedDataSize, the iSER layer at the initiator MUST do the 
following: 


a. It MUST allocate a Write STag for the I/O Buffer defined by 
the qualifier DataDescriptorOut. The DataDescriptorOut 
describes the I/O buffer starting with the immediate 
unsolicited data (if any), followed by the non-immediate 
unsolicited data (if any) and solicited data. This means 
that the BufferOffset for the SCSI Data-out for this 
command is equal to the TO. This implies that a zero TO 
for this STag points to the beginning of this I/O Buffer. 


b. It MUST establish a Local Mapping that associates the 
Initiator Task Tag (ITT) to the Write STag. 


C. It MUST Advertise the Write STag to the target by sending 
it as the Write STag in the iSER header of the iSER Message 
(the payload of the SendSE Message of RCaP) containing the 
SCSI write or bidirectional command PDU. See Section 9.2 
on iSER Header Format for the iSCSI Control-Type PDU. 


For a SCSI read or bidirectional command, the iSCSI layer at the 


initiator MUST invoke the Send Control Operational Primitive 
qualified with DataDescriptorIn, which defines the initiator's I/O 
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Buffer for receiving the SCSI Read data. The iSER layer at the 
initiator MUST do the following: 


a. It MUST allocate a Read STag for the I/O Buffer. 


b. It MUST establish a Local Mapping that associates the 
Initiator Task Tag (ITT) to the Read STag. 


c. It MUST Advertise the Read STag to the target by sending it 
as the Read STag in the iSER header of the iSER Message 
(the payload of the SendSE Message of RCaP) containing the 
SCSI read or bidirectional command PDU. See Section 9.2 on 
iSER Header Format for the iSCSI Control-Type PDU. 


If the amount of unsolicited data to be transferred in a SCSI command 
exceeds TargetRecvDataSegmentLength, then the iSCSI layer at the 
initiator MUST segment the data into multiple iSCSI control-type 
PDUs, with the data segment length in all PDUs generated except the 
last one having exactly the size TargetRecvDataSegmentLength. The 
data segment length of the last iSCSI control-type PDU carrying the 
unsolicited data can be up to TargetRecvDataSegmentLength. 


When the iSER layer at the target receives the SCSI command, it MUST 
establish a Remote Mapping that associates the ITT to the Advertised 
Write STag and the Read STag if present in the iSER header. The 
Write STag is used by the iSER layer at the target in handling the 
data transfer associated with the R2T PDU(s) as described in Section 
7.3.6. The Read STag is used in handling the SCSI Data-in PDU(s) 
from the iSCSI layer at the target as described in Section 7.3.5. 


.2. SCSI Response 
Type: control-type PDU 
PDU-specific qualifiers:  DataDescriptorStatus 


The iSCSI layer at the target MUST invoke the Send_Control 
Operational Primitive qualified with DataDescriptorStatus, which 
defines the buffer containing the sense and response information. 

The iSCSI layer at the target MUST always return the SCSI status for 
a SCSI command in a separate SCSI Response PDU. "Phase collapse" for 
transferring SCSI status in a SCSI Data-in PDU MUST NOT be used. The 
iSER layer at the target sends the SCSI Response PDU according to the 
following rules: 


* If no STags are Advertised by the initiator in the iSER Message 


containing the SCSI command PDU, then the iSER layer at the target 
MUST send a SendSE Message containing the SCSI Response PDU. 
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* If the initiator Advertised a Read STag in the iSER Message 
containing the SCSI Command PDU, then the iSER layer at the target 
MUST send a SendInvSE Message containing the SCSI Response PDU. 
The header of the SendInvSE Message MUST carry the Read STag to be 
invalidated at the initiator. 


* If the initiator Advertised only the Write STag in the iSER 
Message containing the SCSI Command PDU, then the iSER layer at 
the target MUST send a SendInvSE Message containing the SCSI 
Response PDU. The header of the SendInvSE Message MUST carry the 
Write STag to be invalidated at the initiator. 


When the iSCSI layer at the target invokes the Send_Control 
Operational Primitive to send the SCSI Response PDU, the iSER layer 
at the target MUST invalidate the Remote Mapping that associates the 
ITT to the Advertised STag(s) before transferring the SCSI Response 
PDU to the initiator. 


Upon receiving the SendInvSE Message containing the SCSI Response PDU 
from the target, the RCaP layer at the initiator will invalidate the 
STag specified in the header. The iSER layer at the initiator MUST 
ensure that the correct STag is invalidated. If both the Read and 
the Write STags are Advertised earlier by the initiator, then the 
iSER layer at the initiator MUST explicitly invalidate the Write STag 
upon receiving the SendInvSE Message because the header of the 
SendInvSE Message can only carry one STag (in this case, the Read 
STag) to be invalidated. 


The iSER layer at the initiator MUST ensure the invalidation of the 
STag(s) used in a command before notifying the iSCSI layer at the 
initiator by invoking the Control Notify Operational Primitive 
qualified with the SCSI Response. This precludes the possibility of 
using the STag(s) after the completion of the command, thereby 
causing data corruption. 


When the iSER layer at the initiator receives the SendSE or the 
SendInvSE Message containing the SCSI Response PDU, it SHOULD 
invalidate the Local Mapping that associates the ITT to the local 
STag(s). The iSER layer MUST ensure that all local STag(s) 
associated with the ITT are invalidated before notifying the iSCSI 
layer of the SCSI Response PDU by invoking the Control Notify 
Operational Primitive qualified with the SCSI Response PDU. 
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.3. Task Management Function Request/Response 
Type: control-type PDU 


PDU-specific qualifiers (for TMF Request):  DataDescriptorOut, 
DataDescriptorIn 


The iSER layer MUST use a SendSE Message to send the Task Management 
Function Request/Response PDU. 


For the Task Management Function Request with the TASK REASSIGN 
function, the iSER layer at the initiator MUST do the following: 


* It MUST use the ITT as specified in the Referenced Task Tag from 
the Task Management Function Request PDU to locate the existing 
STag(s), if any, in the Local Mapping(s) that associates the ITT 
to the local STag(s). 


* It MUST invalidate the existing STag(s), if any, and the Local 
Mapping(s) that associates the ITT to the local STag(s). 


* It MUST allocate a Read STag for the I/O Buffer as defined by the 
qualifier DataDescriptorIn if the Send Control Operational 
Primitive invocation is qualified with DataDescriptorIn. 


* It MUST allocate a Write STag for the I/O Buffer as defined by the 
qualifier DataDescriptorOut if the Send Control Operational 
Primitive invocation is qualified with DataDescriptorOut. 


* If STags are allocated, it MUST establish a new Local Mapping(s) 
that associate the ITT to the allocated STag(s). 


* It MUST Advertise the STags, if allocated, to the target in the 
iSER header of the SendSE Message carrying the iSCSI PDU, as 
described in Section 9.2. 


For the Task Management Function Request with the TASK REASSIGN 
function for a SCSI read or bidirectional command, the iSCSI layer at 
the initiator MUST set ExpDataSN to 0 since the data transfer and 
acknowledgements happen transparently to the iSCSI layer at the 
initiator. This provides the flexibility to the iSCSI layer at the 
target to request transmission of only the unacknowledged data as 
specified in [RFC3720]. 


When the iSER layer at the target receives the Task Management 


Function Request with the TASK REASSIGN function, it MUST do the 
following: 
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* It MUST use the ITT as specified in the Referenced Task Tag from 
the Task Management Function Request PDU to locate the mappings 
that associate the ITT to the Advertised STag(s) and the local 
STag(s), if any. 


* It MUST invalidate the local STag(s), if any, associated with the 
LTT. 


* It MUST replace the Advertised STag(s) in the Remote Mapping that 
associates the ITT to the Advertised STag(s) with the Write STag 
and the Read STag if present in the iSER header. The Write STag 
is used in the handling of the R2T PDU(s) from the iSCSI layer at 
the target as described in Section 7.3.6. The Read STag is used 
in the handling of the SCSI Data-in PDU(s) from the iSCSI layer at 
the target as described in Section 7.3.5. 


4. SCSI Data-Out 
Type: control-type PDU 
PDU-specific qualifiers:  DataDescriptorOut 


The iSCSI layer at the initiator MUST invoke the Send Control 
Operational Primitive qualified with DataDescriptorOut, which defines 
the initiator's I/O Buffer containing unsolicited SCSI Write data. 


If the amount of unsolicited data to be transferred as SCSI Data-out 
exceeds TargetRecvDataSegmentLength, then the iSCSI layer at the 
initiator MUST segment the data into multiple iSCSI control-type 
PDUs, with the DataSegmentLength having the value of 
TargetRecvDataSegmentLength in all PDUs generated except the last 
one. The DataSegmentLength of the last iSCSI control-type PDU 
carrying the unsolicited data can be up to 
TargetRecvDataSegmentLength. The iSCSI layer at the target MUST 
perform the reassembly function for the unsolicited data. 


For unsolicited data, if the F bit is set to 0 in a SCSI Data-out 
PDU, the iSER layer at the initiator MUST use a Send Message to send 
the SCSI Data-out PDU. If the F bit is set to 1, the iSER layer at 
the initiator MUST use a SendSE Message to send the SCSI Data-out 
PDU. 


Note that for solicited data, the SCSI Data-out PDUs are not used 
Since R2T PDUs are not delivered to the iSCSI layer at the initiator; 
instead, R2T PDUs are transformed by the iSER layer at the target 
into RDMA Read operations. (See Section 7.3.6.) 
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5. SCSI Data-In 
Type: data-type PDU 
PDU-specific qualifiers:  DataDescriptorIn 


When the iSCSI layer at the target is ready to return the SCSI Read 
data to the initiator, it MUST invoke the Put Data Operational 
Primitive qualified with DataDescriptorIn, which defines the SCSI 
Data-in buffer. See Section 7.1 on the general requirement on the 
handling of iSCSI data-type PDUs.  SCSI Data-in PDU(s) are used in 
SCSI Read data transfer as described in Section 9.5.2. 


The iSER layer at the target MUST do the following for each 
invocation of the Put Data Operational Primitive: 


1. It MUST use the ITT in the SCSI Data-in PDU to locate the remote 
Read STag in the Remote Mapping that associates the ITT to 
Advertised STag(s). The Remote Mapping was established earlier 
by the iSER layer at the target when the SCSI read command was 
received from the initiator. 


2. It MUST generate and send an RDMA Write Message containing the 
read data to the initiator. 


a. It MUST use the remote Read STag as the Data Sink STag of the 
RDMA Write Message. 


b. It MUST use the Buffer Offset from the SCSI Data-in PDU as 
the Data Sink Tagged Offset of the RDMA Write Message. 


C. It MUST use DataSegmentLength from the SCSI Data-in PDU to 
determine the amount of data to be sent in the RDMA Write 
Message. 


3. It MUST associate DataSN and ITT from the SCSI Data-in PDU with 
the RDMA Write operation. If the Put Data Operational Primitive 
invocation was qualified with Notify Enable set, then when the 
iSER layer at the target receives a completion from the RCaP 
layer for the RDMA Write Message, the iSER layer at the target 
MUST notify the iSCSI layer by invoking the 
Data Completion Notify Operational Primitive qualified with 
DataSN and ITT. Conversely, if the Put Data Operational 
Primitive invocation was qualified with Notify Enable cleared, 
then the iSER layer at the target MUST NOT notify the iSCSI layer 
on completion and MUST NOT invoke the Data Completion Notify 
Operational Primitive. 
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When the A-bit is set to 1 in the SCSI Data-in PDU, the iSER layer at 
the target MUST notify the iSCSI layer at the target when the data 
transfer is complete at the initiator. To perform this additional 
function, the iSER layer at the target can take advantage of the 
operational ErrorRecoveryLevel if previously disclosed by the iSCSI 
layer via an earlier invocation of the Notice_Key_Values Operational 
Primitive. There are two approaches that can be taken: 


1. If the iSER layer at the target knows that the operational 
ErrorRecoveryLevel is 2, or if the iSER layer at the target does 
not know the operational ErrorRecoveryLevel, then the iSER layer 
at the target MUST issue a zero-length RDMA Read Request Message 
following the RDMA Write Message. When the iSER layer at the 
target receives a completion for the RDMA Read Request Message 
from the RCaP layer, implying that the RDMA-Capable Controller at 
the initiator has completed processing the RDMA Write Message due 
to the completion ordering semantics of RCaP, the iSER layer at 
the target MUST notify the iSCSI layer at the target by invoking 
the Data Ack Notify Operational Primitive qualified with ITT and 
DataSN (see Section 3.2.3). 


2. If the iSER layer at the target knows that the operational 
ErrorRecoveryLevel is 1, then the iSER layer at the target MUST 
do one of the following: 


a. It MUST notify the iSCSI layer at the target by invoking the 
Data_Ack_Notify Operational Primitive qualified with ITT and 
DataSN (see Section 3.2.3) when it receives the local 
completion from the RCaP layer for the RDMA Write Message. 
This is allowed since digest errors do not occur in iSER (see 
Section 10.1.4.2) and a CRC error will cause the connection 
to be terminated and the task to be terminated anyway. The 
local RDMA Write completion from the RCaP layer guarantees 
that the RCaP layer will not access the I/O Buffer again to 
transfer the data associated with that RDMA Write operation. 


b. Alternatively, it MUST use the same procedure for handling 
the data transfer completion at the initiator as for 
ErrorRecoveryLevel 2. 


Note that the iSCSI layer at the target cannot set the A-bit to 1 if 
the ErrorRecoveryLevel=0. 


The SCSI status MUST always be returned in a separate SCSI Response 


PDU. The S bit in the SCSI Data-in PDU MUST always be set to 0. 
There MUST NOT be a "phase collapse" in the SCSI Data-in PDU. 
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Since the RDMA Write Message only transfers the data portion of the 
SCSI Data-in PDU but not the control information in the header, such 
as ExpCmdSN, if timely updates of such information are crucial, the 
iSCSI layer at the initiator MAY issue NOP-Out PDUs to request that 
the iSCSI layer at the target respond with the information using NOP- 
In PDUs. 


6. Ready to Transfer (R2T) 
Type: data-type PDU 
PDU-specific qualifiers:  DataDescriptorOut 


In order to send an R2T PDU, the iSCSI layer at the target MUST 
invoke the Get Data Operational Primitive qualified with 
DataDescriptorOut, which defines the I/O Buffer for receiving the 
SCSI Write data from the initiator. See Section 7.1 on the general 
requirements on the handling of iSCSI data-type PDUs. 


The iSER layer at the target MUST do the following for each 
invocation of the Get Data Operational Primitive: 


1. It MUST ensure a valid local STag for the I/O Buffer and a valid 
Local Mapping that associates the Initiator Task Tag (ITT) to the 
local STag. This may involve allocating a valid local STag and 
establishing a Local Mapping. 


2. It MUST use the ITT in the R2T to locate the remote Write STag in 
the Remote Mapping that associates the ITT to Advertised STag(s). 
The Remote Mapping is established earlier by the iSER layer at 
the target when the iSER Message containing the Advertised Write 
STag and the SCSI Command PDU for a SCSI write or bidirectional 
command is received from the initiator. 


3. If the iSER-ORD value at the target is set to 0, the iSER layer 
at the target MUST terminate the connection and free up the 
resources associated with the connection (as described in Section 
5.2.3) if it receives the R2T PDU from the iSCSI layer at the 
target. Upon termination of the connection, the iSER layer at 
the target MUST notify the iSCSI layer at the target by invoking 
the Connection Terminate Notify Operational Primitive. 


4. If the iSER-ORD value at the target is set to greater than 0, the 
iSER layer at the target MUST transform the R2T PDU into an RDMA 
Read Request Message. While transforming the R2T PDU, the iSER 
layer at the target MUST ensure that the number of outstanding 
RDMA Read Request Messages does not exceed the iSER-ORD value. 

To transform the R2T PDU, the iSER layer at the target: 
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a. MUST derive the local STag and local Tagged Offset from the 
DataDescriptorOut that qualified the Get_Data invocation. 


b. MUST use the local STag as the Data Sink STag of the RDMA 
Read Request Message. 


c. MUST use the local Tagged Offset as the Data Sink Tagged 
Offset of the RDMA Read Request Message. 


d. MUST use the Desired Data Transfer Length from the R2T PDU as 
the RDMA Read Message Size of the RDMA Read Request Message. 


e. MUST use the remote Write STag as the Data Source STag of the 
RDMA Read Request Message. 


f. MUST use the Buffer Offset from the R2T PDU as the Data 
Source Tagged Offset of the RDMA Read Request Message. 


5. It MUST associate R2TSN and ITT from the R2T PDU with the RDMA 
Read operation. If the Get Data Operational Primitive invocation 
is qualified with Notify Enable set, then when the iSER layer at 
the target receives a completion from the RCaP layer for the RDMA 
Read operation, the iSER layer at the target MUST notify the 
iSCSI layer by invoking the Data Completion Notify Operational 
Primitive qualified with R2TSN and ITT. Conversely, if the 
Get Data Operational Primitive invocation is qualified with 
Notify Enable cleared, then the iSER layer at the target MUST NOT 
notify the iSCSI layer on completion and MUST NOT invoke the 
Data Completion Notify Operational Primitive. 


When the RCaP layer at the initiator receives a valid RDMA Read 
Request Message, it will return an RDMA Read Response Message 
containing the solicited write data to the target. When the RCaP 
layer at target receives the RDMA Read Response Message from the 
initiator, it will place the solicited data in the I/O Buffer 
referenced by the Data Sink STag in the RDMA Read Response Message. 


Since the RDMA Read Request Message from the target does not transfer 
the control information in the R2T PDU, such as ExpCmdSN, if timely 
updates of such information are crucial, the iSCSI layer at the 
initiator MAY issue NOP-Out PDUs to request that the iSCSI layer at 
the target respond with the information using NOP-In PDUs. 


Similarly, since the RDMA Read Response Message from the initiator 
only transfers the data but not the control information normally 
found in the SCSI Data-out PDU, such as ExpStatSN, if timely updates 
of such information are crucial, the iSCSI layer at the target MAY 
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issue NOP-In PDUs to request that the iSCSI layer at the initiator 
respond with the information using NOP-Out PDUs. 


7. Asynchronous Message 

Type:  control-type PDU 

PDU-specific qualifiers:  DataDescriptorSense 
The iSCSI layer MUST invoke the Send Control Operational Primitive 
qualified with DataDescriptorSense, which defines the buffer 


containing the sense and iSCSI Event information. The iSER layer 
MUST use a SendSE Message to send the Asynchronous Message PDU. 


8. Text Request and Text Response 
Type: control-type PDU 


PDU-specific qualifiers:  DataDescriptorTextOut (for Text 
Request), DataDescriptorIn (for Text Response) 


The iSCSI layer MUST invoke the Send Control Operational Primitive 
qualified with DataDescriptorTextOut (or DataDescriptorIn), which 
defines the Text Request (or Text Response) buffer. The iSER layer 
MUST use SendSE Messages to send the Text Request (or Text Response 
PDUs). 


9. Login Request and Login Response 


During the login negotiation, the iSCSI layer interacts with the 
transport layer directly and the iSER layer is not involved. See 
Section 5.1 on iSCSI/iSER connection setup. If the underlying 
transport is TCP, the Login Request PDUs and the Login Response PDUs 
are exchanged when the connection between the initiator and the 
target is still in the byte stream mode. 


The iSCSI layer MUST not send a Login Request (or a Login Response) 
PDU during the Full Feature Phase. A Login Request (or a Login 
Response) PDU, if used, MUST be treated as an iSCSI protocol error. 
The iSER layer MAY reject such a PDU from the iSCSI layer with an 
appropriate error code. If a Login Request PDU is received by the 
iSCSI layer at the target, it MUST respond with a Reject PDU with a 
reason code of "protocol error". 
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.10. Logout Request and Logout Response 
Type: control-type PDU 
PDU-specific qualifiers: None 


The iSER layer MUST use a SendSE Message to send the Logout Request 
or Logout Response PDU. Sections 5.2.1 and 5.2.2 describe the 
handling of the Logout Request and the Logout Response at the 
initiator and the target and the interactions between the initiator 
and the target to terminate a connection. 


.11. SNACK Request 


Since HeaderDigest and DataDigest must be negotiated to "None", there 
are no digest errors when the connection is in iSER-assisted mode. 
Also, since RCaP delivers all messages in the order they were sent, 
there are no sequence errors when the connection is in iSER-assisted 
mode. Therefore, the iSCSI layer MUST NOT send SNACK Request PDUs. 

A SNCAK Request PDU, if used, MUST be treated as an iSCSI protocol 
error. The iSER layer MAY reject such a PDU from the iSCSI layer 
with an appropriate error code. If a SNACK Request PDU is received 
by the iSCSI layer at the target, it MUST respond with a Reject PDU 
with a reason code of "protocol error". 


.12. Reject 
Type:  control-type PDU 
PDU-specific qualifiers:  DataDescriptorReject 
The iSCSI layer MUST invoke the Send Control Operational Primitive 


qualified with DataDescriptorReject, which defines the Reject buffer. 
The iSER layer MUST use a SendSE Message to send the Reject PDU. 


7.3.13.  NOP-Out and NOP-In 


Ko, 


Type: control-type PDU 


PDU-specific qualifiers:  DataDescriptorNOPOut (for NOP-Out), 
DataDescriptorNOPIn (for NOP-In) 


The iSCSI layer MUST invoke the Send Control Operational Primitive 
qualified with DataDescriptorNOPOut (or DataDescriptorNOPIn), which 
defines the Ping (or Return Ping) data buffer. The iSER layer MUST 
use SendSE Messages to send the NOP-Out (or NOP-In) PDU. 


et al. Standards Track [Page 51] 


RFC 5046 iSER Specification October 2007 


S.l. 


Sd 


Ko, 


Flow Control and STag Management 
Flow Control for RDMA Send Message Types 


Send Message Types in RCaP are used by the iSER layer to transfer 
iSCSI control-type PDUs. Each Send Message Type in RCaP consumes an 
Untagged Buffer at the Data Sink. However, neither the RCaP layer 
nor the iSER layer provides an explicit flow control mechanism for 
the Send Message Types. Therefore, the iSER layer SHOULD provision 
enough Untagged buffers for handling incoming Send Message Types to 
prevent buffer exhaustion at the RCaP layer. If buffer exhaustion 
occurs, it may result in the termination of the connection. 


An implementation may choose to satisfy the buffer requirement by 
using a common buffer pool shared across multiple connections, with 
usage limits on a per-connection basis and usage limits on the buffer 
pool itself. In such an implementation, exceeding the buffer usage 
limit for a connection or the buffer pool itself may trigger 
interventions from the iSER layer to replenish the buffer pool and/or 
to isolate the connection causing the problem. 


iSER also provides the MaxOutstandingUnexpectedPDUs key to be used by 
the initiator and the target to declare the maximum number of 
outstanding "unexpected" control-type PDUs that it can receive. It 
is intended to allow the receiving side to determine the amount of 
buffer resources needed beyond the normal flow control mechanism 
available in iSCSI. 


The buffer resources required at both the initiator and the target as 
a result of control-type PDUs sent by the initiator is described in 
Section 8.1.1. The buffer resources required at both the initiator 
and target as a result of control-type PDUs sent by the target is 
described in Section 8.1.2. 


1. Flow Control for Control-Type PDUs from the Initiator 


The control-type PDUs that can be sent by an initiator to a target 
can be grouped into the following categories: 


1. Regulated:  Control-type PDUs in this category are regulated by 
the iSCSI CmdSN window mechanism and the immediate flag is not 
set. 


2. Unregulated but Expected:  Control-type PDUs in this category are 


not regulated by the iSCSI CmdSN window mechanism but are 
expected by the target. 
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3. Unregulated and Unexpected:  Control-type PDUs in this category 
are not regulated by the iSCSI CmdSN window mechanism and are 
"unexpected" by the target. 


8.1.1.1. Control-Type PDUs from the Initiator in the Regulated Category 


Control-type PDUs that can be sent by the initiator in this category 
are regulated by the iSCSI CmdSN window mechanism and the immediate 
flag is not set. 


The queuing capacity required of the iSCSI layer at the target is 
described in Section 3.2.2.1 of [RFC3720]. For each of the control- 
type PDUs that can be sent by the initiator in this category, the 
initiator MUST provision for the buffer resources required for the 
corresponding control-type PDU sent as a response from the target. 
The following is a list of the PDUs that can be sent by the initiator 
and the PDUs that are sent by the target in response: 


a. When an initiator sends a SCSI Command PDU, it expects a SCSI 
Response PDU from the target. 


b. When the initiator sends a Task Management Function Request 
PDU, it expects a Task Management Function Response PDU from 


the target. 


c. When the initiator sends a Text Request PDU, it expects a 
Text Response PDU from the target. 


d. When the initiator sends a Logout Request PDU, it expects a 
Logout Response PDU from the target. 


e. When the initiator sends a NOP-Out PDU as a ping request with 


ITT != Oxffffffff and TIT = Oxffffffff, it expects a NOP-In 
PDU from the target with the same ITT and TTT as in the ping 
request. 


The response from the target for any of the PDUs enumerated here may 
alternatively be in the form of a Reject PDU sent instead before the 
task is active, as described in Section 6.3 of [RFC3720]. 


8.1.1.2. Control-Type PDUs from the Initiator in the Unregulated but 
Expected Category 


For the control-type PDUs in the Unregulated but Expected category, 


the amount of buffering resources required at the target can be 
predetermined. The following is a list of the PDUs in this category: 
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a. SCSI Data-out PDUs are used by the initiator to send 
unsolicited data. The amount of buffer resources required by 
the target can be determined using FirstBurstLength. Note 
that SCSI Data-out PDUs are not used for solicited data since 
the R2T PDU that is used for solicitation is transformed into 
RDMA Read operations by the iSER layer at the target. See 
Section 7.3.4. 


b. A NOP-Out PDU with TIT != Oxffffffff is sent as a ping 
response by the initiator to the NOP-In PDU sent as a ping 
request by the target. 


8.1.1.3. Control-Type PDUs from the Initiator in the Unregulated and 


Ko, 


Unexpected Category 


PDUs in the Unregulated and Unexpected category are PDUs with the 
immediate flag set. The number of PDUs in this category that can be 
sent by an initiator is controlled by the value of 
MaxOutstandingUnexpectedPDUs declared by the target (see Section 
6.7). After a PDU in this category is sent by the initiator, it is 
outstanding until it is retired. At any time, the number of 
outstanding unexpected PDUs MUST not exceed the value of 
MaxOutstandingUnexpectedPDUs declared by the target. 


The target uses the value of MaxOutstandingUnexpectedPDUs that it 
declared to determine the amount of buffer resources required for 
control-type PDUs in this category that can be sent by an initiator. 
For the initiator, for each of the control-type PDUs that can be sent 
in this category, the initiator MUST provision for the buffer 
resources if required for the corresponding control-type PDU that can 
be sent as a response from the target. 


An outstanding PDU in this category is retired as follows. If the 
CmdSN of the PDU sent by the initiator in this category is x, the PDU 
is outstanding until the initiator sends a non-immediate control-type 
PDU on the same connection with CmdSN = y (where y is at least x) and 
the target responds with a control-type PDU on any connection where 
ExpCmdSN is at least ytl. 


When the number of outstanding unexpected control-type PDUs equals 
MaxOutstandingUnexpectedPDUs, the iSCSI layer at the initiator MUST 
NOT generate any unexpected PDUs that otherwise it would have 
generated, even if it is intended for immediate delivery. 
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.2. Flow Control for Control-Type PDUs from the Target 


Control-type PDUs that can be sent by a target and are expected by 
the initiator are listed in the Regulated category (see Section 
Be dd 


For the control-type PDUs that can be sent by a target and are 
unexpected by the initiator, the number is controlled by 
MaxOutstandingUnexpectedPDUs declared by the initiator (see Section 
6.7). After a PDU in this category is sent by a target, it is 
outstanding until it is retired. At any time, the number of 
outstanding unexpected PDUs MUST not exceed the value of 
MaxOutstandingUnexpectedPDUs declared by the initiator. The 
initiator uses the value of MaxOutstandingUnexpectedPDUs that it 
declared to determine the amount of buffer resources required for 
control-type PDUs in this category that can be sent by a target. The 
following is a list of the PDUs in this category and the conditions 
for retiring the outstanding PDU: 


a. For an Asynchronous Message PDU with StatSN = x, the PDU is 
outstanding until the initiator sends a control-type PDU with 
ExpStatSN set to at least x+1. 


b. For a Reject PDU with StatSN = x that is sent after a task is 
active, the PDU is outstanding until the initiator sends a 
control-type PDU with ExpStatSN set to at least x+1. 


c. For a NOP-In PDU with ITT = Oxffffffff and StatSN = x, the 
PDU is outstanding until the initiator responds with a 
control-type PDU on the same connection where ExpStatSN is at 
least x+1. But if the NOP-In PDU is sent as a ping request 
with TTT != Oxffffffff, the PDU can also be retired when the 
initiator sends a NOP-Out PDU with the same ITT and TTT as in 
the ping request. Note that when a target sends a NOP-In PDU 
as a ping request, it must provision a buffer for the NOP-Out 
PDU sent as a ping response from the initiator. 


When the number of outstanding unexpected control-type PDUs equals 
MaxOutstandingUnexpectedPDUs, the iSCSI layer at the target MUST NOT 
generate any unexpected PDUs that otherwise it would have generated, 
even if its intent is to indicate an iSCSI error condition (e.g., 
Asynchronous Message, Reject). Task timeouts, as in the initiator 
waiting for a command completion or other connection and session 
level exceptions, will ensure that correct operational behavior will 
result in these cases despite not generating the PDU. This rule 
overrides any other requirements elsewhere that require that a Reject 
PDU MUST be sent. 
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(Implementation note: ASCSI task timeout and recovery can be a 
lengthy process and hence SHOULD be avoided by proper provisioning of 
resources.) 


(Implementation note: To ensure that the initiator has a means to 
inform the target that outstanding PDUs have been retired, the target 
should reserve the last unexpected control-type PDU allowable by the 
value of MaxOutstandingUnexpectedPDUs declared by the initiator for 
sending a NOP-In ping request with TTT != Oxffffffff to allow the 
initiator to return the NOP-Out ping response with the current 
ExpStatSN.) 


Flow Control for RDMA Read Resources 


The total number of RDMA Read operations that can be active 
simultaneously on an iSCSI/iSER connection depends on the amount of 
resources allocated as declared in the iSER Hello exchange described 
in Section 5.1.3.  Exceeding the number of RDMA Read operations 
allowed on a connection will result in the connection being 
terminated by the RCaP layer. The iSER layer at the target maintains 
the iSER-ORD to keep track of the maximum number of RDMA Read 
Requests that can be issued by the iSER layer on a particular RCaP 
Stream. 


During connection setup (see Section 5.1), iSER-IRD is known at the 
initiator and iSER-ORD is known at the target after the iSER layers 
at the initiator and the target have respectively allocated the 
connection resources necessary to support RCaP, as directed by the 
Allocate Connection Resources Operational Primitive from the iSCSI 
layer before the end of the iSCSI Login Phase. In the Full Feature 
Phase, the first message sent by the initiator is the iSER Hello 
Message (see Section 9.3), which contains the value of iSER-IRD. In 
response to the iSER Hello Message, the target sends the iSER 
HelloReply Message (see Section 9.4), which contains the value of 
iSER-ORD. The iSER layer at both the initiator and the target MAY 
adjust (lower) the resources associated with iSER-IRD and iSER-ORD 
respectively to match the iSER-ORD value declared in the HelloReply 
Message. The iSER layer at the target MUST flow control the RDMA 
Read Request Messages to not exceed the iSER-ORD value at the target. 


STag Management 


An STag, as defined in [RDMAP], is an identifier of a Tagged Buffer 
used in an RDMA operation. The allocation and the subsequent 
invalidation of the STags are specified in this document if the STags 
are exposed on the wire by being Advertised in the iSER header or 
declared in the header of an RCaP Message. 
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1. Allocation of STags 


When the iSCSI layer at the initiator invokes the Send Control 
Operational Primitive to request that the iSER layer at the initiator 
process a SCSI command, zero, one, or two STags may be allocated by 
the iSER layer. See Section 7.3.1 for details. The number of STags 
allocated depends on whether the command is unidirectional or 
bidirectional and whether or not solicited write data transfer is 
involved. 


When the iSCSI layer at the initiator invokes the Send Control 
Operational Primitive to request that the iSER layer at the initiator 
process a Task Management Function Request with the TASK REASSIGN 
function, besides allocating zero, one, or two STags, the iSER layer 
MUST invalidate the existing STags, if any, associated with the ITT. 
See Section 7.3.3 for details. 


The iSER layer at the target allocates a local Data Sink STag when 
the iSCSI layer at the target invokes the Get Data Operational 
Primitive to request that the iSER layer process an R2T PDU. See 
Section 7.3.6 for details. 


.2.  Invalidation of STags 


The invalidation of the STags at the initiator at the completion of a 
unidirectional or bidirectional command when the associated SCSI 
Response PDU is sent by the target is described in Section 7.3.2. 


When a unidirectional or bidirectional command concludes without the 
associated SCSI Response PDU being sent by the target, the iSCSI 
layer at the initiator MUST request that the iSER layer at the 
initiator invalidate the STags by invoking the 
Deallocate Task Resources Operational Primitive qualified with ITT. 
In response, the iSER layer at the initiator MUST locate the STag(s) 
(if any) in the Local Mapping that associates the ITT to the local 
STag(s). The iSER layer at the initiator MUST invalidate the STag(s) 
(if any) and the Local Mapping. 


For an RDMA Read operation used to realize a SCSI Write data 
transfer, the iSER layer at the target SHOULD invalidate the Data 
Sink STag at the conclusion of the RDMA Read operation referencing 
the Data Sink STag (to permit the immediate reuse of buffer 
resources). 


For an RDMA Write operation used to realize a SCSI Read data 
transfer, the Data Source STag at the target is not declared to the 
initiator and is not exposed on the wire.  Invalidation of the STag 
is thus not specified. 
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When a unidirectional or bidirectional command concludes without the 
associated SCSI Response PDU being sent by the target, the iSCSI 
layer at the target MUST request that the iSER layer at the target 
invalidate the STags by invoking the Deallocate Task Resources 


Operational Primitive qualified with ITT. In response, the iSER 
layer at the target MUST locate the local STag(s) (if any) in the 
Local Mapping that associates the ITT to the local STag(s). The iSER 


layer at the target MUST invalidate the local STag(s) (if any) and 
the mapping. 


iSER Control and Data Transfer 


For iSCSI data-type PDUs (see Section 7.1), the iSER layer uses RDMA 
Read and RDMA Write operations to transfer the solicited data. For 
iSCSI control-type PDUs (see Section 7.2), the iSER layer uses Send 
Message Types of RCaP. 


iSER Header Format 


An iSER header MUST be present in every Send Message Type of RCaP. 
The iSER header is located in the first 12 bytes of the message 
payload of the Send Message Type of RCaP, as shown in Figure 2. 


0 1 2 3 
Qd 2 3475c0.71.-8 90 Fr2—3 4 5.6 17.8.9 0.t 2.25 6.7.8 9-01 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 

| Opcode | Opcode Specific Fields 

十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Opcode Specific Fields 

十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Opcode Specific Fields 

十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Figure 2. iSER Header Format 
Opcode - Operation Code: 4 bits 
The Opcode field identifies the type of iSER Messages: 
0001b = iSCSI control-type PDU 


0010b 


iSER Hello Message 
0011b = iSER HelloReply Message 


All other opcodes are reserved. 
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iSER Header Format for the iSCSI Control-Type PDU 


iSER layer uses Send Message Types of RCaP to transfer iSCSI 
trol-type PDUs (see Section 7.2). The message payload of each of 

Send Message Types of RCaP used for transferring an iSER Message 
tains an iSER Header followed by an iSCSI control-type PDU. 


iSER header in a Send Message Type of RCaP carrying an iSCSI 
trol-type PDU MUST have the format as described in Figure 3. 


0 1 2 3 
QE 72. 3:04. 57:6: 178790 T .23 5,.-6. T 8 907 1.2.3: 4.5. 6, 7 78:9 Qc T 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 

Iw |&| | 
| 0001b |s|s| Reserved | 
| [vivi | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Write STag (or N/A) | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Read STag (or N/A) | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Figure 3. iSER Header Format for iSCSI Control-Type PDU 
- Write STag Valid flag: 1 bit 


This flag indicates the validity of the Write STag field of the 
iSER Header. If set to one, the Write STag field in this iSER 
Header is valid. If set to zero, the Write STag field in this 
iSER Header MUST be ignored at the receiver. The Write STag 
Valid flag is set to one when there is solicited data to be 
transferred for a SCSI write or bidirectional command, or when 
there are non-immediate unsolicited and solicited data to be 
transferred for the referenced task specified in a Task 
Management Function Request with the TASK REASSIGN function. 


- Read STag Valid flag: 1 bit 


This flag indicates the validity of the Read STag field of the 
iSER Header. If set to one, the Read STag field in this iSER 
Header is valid. If set to zero, the Read STag field in this 
iSER Header MUST be ignored at the receiver. The Read STag Valid 
flag is set to one for a SCSI read or bidirectional command, or 
for a Task Management Function Request with the TASK REASSIGN 
function. 
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Write STag - Write Steering Tag: 32 bits 


This field contains the Write STag when the Write STag Valid flag 
is set to one. For a SCSI write or bidirectional command, the 
Write STag is used to Advertise the initiator's I/O Buffer 
containing the solicited data. For a Task Management Function 
Request with the TASK REASSIGN function, the Write STag is used 
to Advertise the initiator's I/O Buffer containing the non- 
immediate unsolicited data and solicited data. This Write STag 
is used as the Data Source STag in the resultant RDMA Read 
operation(s). When the Write STag Valid flag is set to zero, 
this field MUST be set to zero. 


Read STag - Read Steering Tag: 32 bits 


This field contains the Read STag when the Read STag Valid flag 
is set to one. The Read STag is used to Advertise the 
initiator's Read I/O Buffer of a SCSI read or bidirectional 
command, or of a Task Management Function Request with the TASK 
REASSIGN function. This Read STag is used as the Data Sink STag 
in the resultant RDMA Write operation(s). When the Read STag 
Valid flag is zero, this field MUST be set to zero. 


Reserved: 


Reserved fields MUST be set to zero on transmit and MUST be 
ignored on reception. 


iSER Header Format for the iSER Hello Message 


An iSER Hello Message MUST only contain the iSER header, which MUST 
have the format as described in Figure 4. The iSER Hello Message is 
the first iSER Message sent on the RCaP Stream from the iSER layer at 
the initiator to the iSER layer at the target. 


0 1 2 3 
0123 4.5.6 7:89 0 L2 3.4 5967 8.9.0 12 34 5.6 7.8.9 0.1 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


0010b Rsvd MaxVer| MinVer iSER-IRD 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Reserved | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Reserved | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Figure 4. iSER Header Format for iSER Hello Message 
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MaxVer - Maximum Version: 4 bits 
This field specifies the maximum version of the iSER protocol 
supported. It MUST be set to one to indicate the version of the 
specification described in this document. 

MinVer - Minimum Version: 4 bits 
This field specifies the minimum version of the iSER protocol 
supported. It MUST be set to one to indicate the version of the 
specification described in this document. 

iSER-IRD: 16 bits 
This field contains the value of the iSER-IRD at the initiator. 


Reserved (Rsvd): 


Reserved fields MUST be set to zero on transmit, and MUST be 
ignored on reception. 


9.4.  iSER Header Format for the iSER HelloReply Message 


An iSER HelloReply Message MUST only contain the iSER header which 

MUST have the format as described in Figure 5. The iSER HelloReply 
Message is the first iSER Message sent on the RCaP Stream from the 

iSER layer at the target to the iSER layer at the initiator. 


0 1 2 3 
0127234507839 0123 45-67 8 901223 45 67.89 01 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 

[R] | | | 
| 0011b |Rsvd |E| MaxVer| CurVer| iSER-ORD | 
| | IM | | | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Reserved | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Reserved | 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
Figure 5. iSER Header Format for iSER HelloReply Message 
REJ - Reject flag: 1 bit 
This flag indicates whether the target is rejecting this 


connection. If set to one, the target is rejecting the 
connection. 
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MaxVer - Maximum Version: 4 bits 


This field specifies the maximum version of the iSER protocol 
supported. It MUST be set to one to indicate the version of the 
specification described in this document. 


CurVer - Current Version: 4 bits 


This field specifies the current version of the iSER protocol 
supported. It MUST be set to one to indicate the version of the 
Specification described in this document. 


iSER-ORD: 16 bits 
This field contains the value of the iSER-ORD at the target. 
Reserved (Rsvd): 


Reserved fields MUST be set to zero on transmit, and MUST be 
ignored on reception. 


9.5. SCSI Data Transfer Operations 


The iSER layer at the initiator and the iSER layer at the target 
handle each SCSI Write, SCSI Read, and bidirectional operation as 
described below. 


9.5.1. SCSI Write Operation 


The iSCSI layer at the initiator MUST invoke the Send Control 
Operational Primitive to request that the iSER layer at the initiator 
send the SCSI write command. The iSER layer at the initiator MUST 
request that the RCaP layer transmit a SendSE Message with the 
message payload consisting of the iSER header followed by the SCSI 
Command PDU and immediate data (if any). If there is solicited data, 
the iSER layer MUST Advertise the Write STag in the iSER header of 
the SendSE Message, as described in Section 9.2. Upon receiving the 
SendSE Message, the iSER layer at the target MUST notify the iSCSI 
layer at the target by invoking the Control Notify Operational 
Primitive qualified with the SCSI Command PDU. See Section 7.3.1 for 
details on the handling of the SCSI write command. 


For the non-immediate unsolicited data, the iSCSI layer at the 
initiator MUST invoke a Send Control Operational Primitive qualified 
with the SCSI Data-out PDU. Upon receiving each Send or SendSE 
Message containing the non-immediate unsolicited data, the iSER layer 
at the target MUST notify the iSCSI layer at the target by invoking 
the Control Notify Operational Primitive qualified with the SCSI 
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Data-out PDU. See Section 7.3.4 for details on the handling of the 
SCSI Data-out PDU. 


For the solicited data, when the iSCSI layer at the target has an I/O 
Buffer available, it MUST invoke the Get_Data Operational Primitive 
qualified with the R2T PDU. See Section 7.3.6 for details on the 
handling of the R2T PDU. 


When the data transfer associated with this SCSI Write operation is 
complete, the iSCSI layer at the target MUST invoke the Send_Control 
Operational Primitive when it is ready to send the SCSI Response PDU. 
Upon receiving a SendSE or SendInvSE Message containing the SCSI 
Response PDU, the iSER layer at the initiator MUST notify the iSCSI 
layer at the initiator by invoking the Control_Notify Operational 
Primitive qualified with the SCSI Response PDU. See Section 7.3.2 
for details on the handling of the SCSI Response PDU. 


.2. SCSI Read Operation 


The iSCSI layer at the initiator MUST invoke the Send_Control 
Operational Primitive to request that the iSER layer at the initiator 
to send the SCSI read command. The iSER layer at the initiator MUST 
request that the RCaP layer transmit a SendSE Message with the 
message payload consisting of the iSER header followed by the SCSI 
Command PDU. The iSER layer at the initiator MUST Advertise the Read 
STag in the iSER header of the SendSE Message, as described in 
Section 9.2. Upon receiving the SendSE Message, the iSER layer at 
the target MUST notify the iSCSI layer at the target by invoking the 
Control_Notify Operational Primitive qualified with the SCSI Command 
PDU. See Section 7.3.1 for details on the handling of the SCSI read 
command. 


When the requested SCSI data is available in the I/O Buffer, the 
iSCSI layer at the target MUST invoke the Put_Data Operational 
Primitive qualified with the SCSI Data-in PDU. See Section 7.3.5 for 
details on the handling of the SCSI Data-in PDU. 


When the data transfer associated with this SCSI Read operation is 
complete, the iSCSI layer at the target MUST invoke the Send_Control 
Operational Primitive when it is ready to send the SCSI Response PDU. 
Upon receiving the SendInvSE Message containing the SCSI Response 
PDU, the iSER layer at the initiator MUST notify the iSCSI layer at 
the initiator by invoking the Control_Notify Operational Primitive 
qualified with the SCSI Response PDU. See Section 7.3.2 for details 
on the handling of the SCSI Response PDU. 
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9.5.3. Bidirectional Operation 


The initiator and the target handle the SCSI Write and the SCSI Read 
portions of this bidirectional operation the same as described in 
Sections 9.5.1 and 9.5.2, respectively. 


10. iSER Error Handling and Recovery 


RCaP provides the iSER layer with reliable in-order delivery. 
Therefore, the error management needs of an iSER-assisted connection 
are somewhat different than those of a Traditional iSCSI connection. 


10.1. Error Handling 


iSER error handling is described in the following sections, 
classified loosely based on the sources of errors: 


1. Those originating at the transport layer (e.g., TCP). 
2. Those originating at the RCaP layer. 
3. Those originating at the iSER layer. 
4. Those originating at the iSCSI layer. 
10.1.1. Errors in the Transport Layer 


If the transport layer is TCP, then TCP packets with detected errors 
are silently dropped by the TCP layer and result in retransmission at 
the TCP layer. This has no impact on the iSER layer. However, 
connection loss (e.g., link failure) and unexpected termination 
(e.g., TCP graceful or abnormal close without the iSCSI Logout 
exchanges) at the transport layer will cause the iSCSI/iSER 
connection to be terminated as well. 


10.1.1.1. Failure in the Transport Layer before RCaP Mode Is Enabled 


If the connection is lost or terminated before the iSCSI layer 
invokes the Allocate_Connection_Resources Operational Primitive, the 
login process is terminated and no further action is required. 


If the connection is lost or terminated after the iSCSI layer has 
invoked the Allocate_Connection_Resources Operational Primitive, then 
the iSCSI layer MUST request that the iSER layer deallocate all 
connection resources by invoking the Deallocate_Connection_Resources 
Operational Primitive. 
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1.1.2. Failure in the Transport Layer after RCaP Mode Is Enabled 


If the connection is lost or terminated after the iSCSI layer has 
invoked the Enable_Datamover Operational Primitive, the iSER layer 
MUST notify the iSCSI layer of the connection loss by invoking the 
Connection_Terminate_Notify Operational Primitive. Prior to invoking 
the Connection_Terminate_Notify Operational Primitive, the iSER layer 
MUST perform the actions described in Section 5.2.3.2. 


1.2. Errors in the RCaP Layer 


The RCaP layer does not have error recovery operations built in. If 
errors are detected at the RCaP layer, the RCaP layer will terminate 
the RCaP Stream and the associated connection. 


1.2.1. Errors Detected in the Local RCaP Layer 


If an error is encountered at the local RCaP layer, the RCaP layer 
MAY send a Terminate Message to the Remote Peer to report the error 
if possible. (For iWARP, see [RDMAP] for the list of errors where a 
Terminate Message is sent.) The RCaP layer is responsible for 
terminating the connection. After the RCaP layer notifies the iSER 
layer that the connection is terminated, the iSER layer MUST notify 
the iSCSI layer by invoking the Connection_Terminate_Notify 
Operational Primitive. Prior to invoking the 
Connection_Terminate_Notify Operational Primitive, the iSER layer 
MUST perform the actions described in Section 5.2.3.2. 


1.2.2. Errors Detected in the RCaP Layer at the Remote Peer 


If an error is encountered at the RCaP layer at the Remote Peer, the 
RCaP layer at the Remote Peer may send a Terminate Message to report 
the error if possible. If it is unable to send the Terminate 
Message, the connection is terminated. This is treated the same as a 
failure in the transport layer after RDMA is enabled as described in 
Section. 10.1.1.2. 


If an error is encountered at the RCaP layer at the Remote Peer and 
it is able to send a Terminate Message, the RCaP layer at the Remote 
Peer is responsible for terminating the connection. After the local 
RCaP layer notifies the iSER layer that the connection is terminated, 
the iSER layer MUST notify the iSCSI layer by invoking the 
Connection_Terminate_Notify Operational Primitive. Prior to invoking 
the Connection_Terminate_Notify Operational Primitive, the iSER layer 
MUST perform the actions described in Section 5.2.3.2. 
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10.1.3. Errors in the iSER Layer 


The error handling due to errors at the iSER layer is described in 
the following sections. 


10.1.3.1. Insufficient Connection Resources to Support RCaP at 
Connection Setup 


After the iSCSI layer at the initiator invokes the 
Allocate_Connection_Resources Operational Primitive during the iSCSI 
Login Negotiation Phase, if the iSER layer at the initiator fails to 
allocate the connection resources necessary to support RCaP, it MUST 
return a status of failure to the iSCSI layer at the initiator. The 
iSCSI layer at the initiator MUST terminate the connection as 
described in Section 5.2.3.1. 


After the iSCSI layer at the target invokes the 
Allocate_Connection_Resources Operational Primitive during the iSCSI 
Login Negotiation Phase, if the iSER layer at the target fails to 
allocate the connection resources necessary to support RCaP, it MUST 
return a status of failure to the iSCSI layer at the target. The 
iSCSI layer at the target MUST send a Login Response with a status 
class of 3 (Target Error), and a status code of "0302" (Out of 
Resources). The iSCSI layers at the initiator and the target MUST 
terminate the connection as described in Section 5.2.3.1. 


10.1.3.2.  iSER Negotiation Failures 


If the RCaP or iSER related parameters declared by the initiator in 
the iSER Hello Message are unacceptable to the iSER layer at the 
target, the iSER layer at the target MUST set the Reject (REJ) flag, 
as described in Section 9.4, in the iSER HelloReply Message. The 
following are the cases when the iSER layer MUST set the REJ flag to 
one in the HelloReply Message: 


* The initiator-declared iSER-IRD value is greater than 0 and the 
target-declared iSER-ORD value is 0. 


* The initiator-supported and the target-supported iSER protocol 
versions do not overlap. 


After requesting that the RCaP layer send the iSER HelloReply 


Message, the handling of the error situation is the same as that for 
iSER format errors as described in Section 10.1.3.3. 
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1.3.3.  iSER Format Errors 


The following types of errors in an iSER header are considered format 
errors: 


* Illegal contents of any iSER header field 
* Inconsistent field contents in an iSER header 


* Length error for an iSER Hello or HelloReply Message (see Section 
9.3 and 9.4) 


When a format error is detected, the following events MUST occur in 
the specified sequence: 


1. The iSER layer MUST request that the RCaP layer terminate the 
RCaP Stream. The RCaP layer MUST terminate the associated 
connection. 


2. The iSER layer MUST notify the iSCSI layer of the connection 
termination by invoking the Connection_Terminate_Notify 
Operational Primitive. Prior to invoking the 
Connection_Terminate_Notify Operational Primitive, the iSER layer 
MUST perform the actions described in Section 5.2.3.2. 


1.3.4.  iSER Protocol Errors 


The first iSER Message sent by the iSER layer at the initiator after 
transitioning into iSER-assisted mode MUST be the iSER Hello Message 
(see Section 9.3). Likewise, the first iSER Message sent by the iSER 
layer at the target after transitioning into iSER-assisted mode MUST 
be the iSER HelloReply Message (see Section 9.4). Failure to send 
the iSER Hello or HelloReply Message, as indicated by the wrong 
Opcode in the iSER header, is a protocol error. The handling of this 
error situation is the same as that for iSER format errors as 
described in Section 10.1.3.3. 


If the sending side of an iSER-enabled connection acts in a manner 
not permitted by the negotiated or declared login/text operational 
key values as described in Section 6, this is a protocol error, and 
the receiving side MAY handle this the same as for iSER format errors 
as described in Section 10.1.3.3. 


1.4. Errors in the iSCSI Layer 


The error handling due to errors at the iSCSI layer is described in 
the following sections. For error recovery, see Section 10.2. 
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1.4.1. iSCSI Format Errors 


When an iSCSI format error is detected, the iSCSI layer MUST request 
that the iSER layer terminate the RCaP Stream by invoking the 
Connection_Terminate Operational Primitive. For more details on the 
connection termination, see Section 5.2.3.1. 


1.4.2. iSCSI Digest Errors 


In the iSER-assisted mode, the iSCSI layer will not see any digest 
error because both the HeaderDigest and the DataDigest keys are 
negotiated to "None". 


1.4.3. iSCSI Sequence Errors 


For Traditional iSCSI, sequence errors are caused by dropped PDUs due 
to header or data digest errors. Since digests are not used in 
iSER-assisted mode and the RCaP layer will deliver all messages in 
the order they were sent, sequence errors will not occur in iSER- 
assisted mode. 


1.4.4. iSCSI Protocol Error 

When the iSCSI layer handles certain protocol errors by dropping the 
connection, the error handling is the same as that for iSCSI format 
errors as described in Section 10.1.4.1. 

When the iSCSI layer uses the iSCSI Reject PDU and response codes to 
handle certain other protocol errors, no special handling at the iSER 
layer is required. 


1.4.5. SCSI Timeouts and Session Errors 


SCSI Timeouts and Session Errors are handled at the iSCSI layer and 
no special handling at the iSER layer is required. 


1.4.6. iSCSI Negotiation Failures 


For negotiation failures that happen during the Login Phase at the 
initiator after the iSCSI layer has invoked the 
Allocate_Connection_Resources Operational Primitive and before the 
Enable Datamover Operational Primitive has been invoked, the iSCSI 
layer MUST request that the iSER layer deallocate all connection 
resources by invoking the Deallocate_Connection_Resources Operational 
Primitive. The iSCSI layer at the initiator MUST terminate the 
connection. 
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For negotiation failures during the Login Phase at the target, the 
iSCSI layer can use a Login Response with a status class other than 0 
(success) to terminate the Login Phase. If the iSCSI layer has 
invoked the Allocate_Connection_Resources Operational Primitive 
before the Enable Datamover Operational Primitive has been invoked, 
the iSCSI layer at the target MUST request that the iSER layer at the 
target deallocate all connection resources by invoking the 

Deallocate Connection Resources Operational Primitive. The iSCSI 
layer at both the initiator and the target MUST terminate the 
connection. 


During the iSCSI Login Phase, if the iSCSI layer at the initiator 
receives a Login Response from the target with a status class other 
than 0 (Success) after the iSCSI layer at the initiator has invoked 
the Allocate Connection Resources Operational Primitive, the iSCSI 
layer MUST request the iSER layer to deallocate all connection 
resources by invoking the Deallocate Connection Resources Operational 
Primitive. The iSCSI layer MUST terminate the connection in this 
case. 


For negotiation failures during the Full Feature Phase, the error 
handling is left to the iSCSI layer and no special handling at the 
iSER layer is required. 


.2. Error Recovery 


Error recovery requirements of iSCSI/iSER are the same as that of 
Traditional iSCSI. All three ErrorRecoveryLevels as defined in 
[RFC3720] are supported in iSCSI/iSER. 


* For ErrorRecoveryLevel 0, session recovery is handled by iSCSI and 
no special handling by the iSER layer is required. 


* For ErrorRecoveryLevel 1, see Section 10.2.1 on PDU Recovery. 


* For ErrorRecoveryLevel 2, see Section 10.2.2 on Connection 
Recovery. 


The iSCSI layer may invoke the Notice Key Values Operational 
Primitive during connection setup to request that the iSER layer take 
note of the value of the operational ErrorRecoveryLevel, as described 
in Sections 5.1.1 and 5.1.2. 


.2.1. PDU Recovery 


As described in Sections 10.1.4.2 and 10.1.4.3, digest and sequence 
errors will not occur in the iSER-assisted mode. If the RCaP layer 
detects an error, it will close the iSCSI/iSER connection, as 
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described in Section 10.1.2. Therefore, PDU recovery is not useful 
in the iSER-assisted mode. 


The iSCSI layer at the initiator SHOULD disable iSCSI timeout-driven 
PDU retransmissions. 


-2.2. Connection Recovery 


The iSCSI layer at the initiator MAY reassign connection allegiance 
for non-immediate commands that are still in progress and are 
associated with the failed connection by using a Task Management 
Function Request with the TASK REASSIGN function. See Section 7.3.3 
for more details. 


When the iSCSI layer at the initiator does a task reassignment for a 
SCSI write command, it MUST qualify the Send_Control Operational 
Primitive invocation with DataDescriptorOut, which defines the I/O 
Buffer for both the non-immediate unsolicited data and the solicited 
data. This allows the iSCSI layer at the target to use recovery R2Ts 
to request data originally sent as unsolicited and solicited from the 
initiator. 


When the iSCSI layer at the target accepts a reassignment request for 
a SCSI read command, it MUST request that the iSER layer process SCSI 
Data-in for all unacknowledged data by invoking the Put_Data 
Operational Primitive. See Section 7.3.5 on the handling of SCSI 
Data-in. 


When the iSCSI layer at the target accepts a reassignment request for 
a SCSI write command, it MUST request that the iSER layer process a 
recovery R2T for any non-immediate unsolicited data and any solicited 
data sequences that have not been received by invoking the Get_Data 
Operational Primitive. See Section 7.3.6 on the handling of Ready To 
Transfer (R2T). 


The iSCSI layer at the target MUST NOT issue recovery R2Ts on an 
iSCSI/iSER connection for a task for which the connection allegiance 
was never reassigned. The iSER layer at the target MAY reject such a 
recovery R2T received via the Get_Data Operational Primitive 
invocation from the iSCSI layer at the target, with an appropriate 
error code. 


The iSER layer at the target will process the requests invoked by the 


Put_Data and Get_Data Operational Primitives for a reassigned task in 
the same way as for the original commands. 
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Security Considerations 


When iSER is layered on top of an RCaP layer and provides the RDMA 
extensions to the iSCSI protocol, the security considerations of iSER 
are the same as that of the underlying RCaP layer. For iWARP, this 
is described in [RDMAP] and [RDDPSEC]. 


Since the iSER-assisted iSCSI protocol is still functionally iSCSI 
from a security considerations perspective, all of the iSCSI security 
requirements as described in [RFC3720] and [RFC3723] apply. If the 
IPsec [IPSEC] mechanism is used, then it MUST be established before 
the connection transitions to the iSER-assisted mode. If iSER is 
layered on top of a non-IP based RCaP layer, all the security 
protocol mechanisms applicable to that RCaP layer are also applicable 
to an iSCSI/iSER connection. If iSER is layered on top of a non-IP 
protocol, the IPsec mechanism as specified in [RFC3720] MUST be 
implemented at any point where the iSER protocol enters the IP 
network (e.g., via gateways), and the non-IP protocol SHOULD 
implement (optional to use) a packet-by packet security protocol 
equal in strength to the IPsec mechanism specified by [RFC3720]. 


To minimize the potential for a denial-of-service attack, the iSCSI 
layer MUST NOT request that the iSER layer allocate the connection 
resources necessary to support RCaP until the iSCSI layer is 
sufficiently far along in the iSCSI Login Phase that it is reasonably 
certain that the peer side is not an attacker, as described in 
Sections 5.1.1 and 5.1.2. 


Note that the IPsec requirements for this document are based on the 
version of IPsec specified in RFC 2401 [IPSEC] and related RFCs, as 
profiled by RFC 3723 [RFC3723], despite the existence of a newer 
version of IPsec specified in RFC 4301 [RFC4301] and related RFCs. 
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Appendix A. iWARP Message Format for iSER 


This section is for information only and is NOT part of the standard. 
It simply depicts the iWARP Message format for the various iSER 
Messages when the transport layer is TCP. 


A.1. iWARP Message Format for iSER Hello Message 


The following figure depicts an iSER Hello Message encapsulated in an 
iWARP SendSE Message. 


0 1 2 3 
0.12.3.4 5.6 79.9 0 1.23. 4 5.6 7. 0-9 0 L2 3-4 5.6 7.8.9 0-41 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| MPA Header | DDP Control | RDMA Control | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Reserved | 
T—R—4—4-—4-—4-t-t-t--t---*-t-t-t-t-—t-—L-L-4— 9 ——4-4-4-4-4-4-4-4 
| (Send) Queue Number | 
T—R—4—4-—4-—.-t-t-t-t----t-t-t-t-t-—t—L-—L-4— ——4—4-4-4-4-4-4-4-4 

| (Send) Message Sequence Number 
T—R—4—4-—4-4-t-t-t-t----t-t-t-t-t-—t—L-— 4-4 ———4—4-4-4-4-4-4-4 
(Send) Message Offset | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
0010b | Zeros | 0001b | 0001b | iSER-IRD | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


2 

| All Zeros | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| All Zeros | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| MPA CRC | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Figure 6. SendSE Message Containing an iSER Hello Message 
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A.2.  iWARP Message Format for iSER HelloReply Message 


The following figure depicts an iSER HelloReply Message encapsulated 
in an iWARP SendSE Message. The Reject (REJ) flag is set to O. 


0 I 2 3 
012345678901234567829012345678901 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| MPA Header | DDP Control | RDMA Control | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Reserved | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| (Send) Queue Number | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 

| (Send) Message Sequence Number 

十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| (Send) Message Offset | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| 0011b |Zeros|0| 0001b | 0001b | iSER-ORD | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| All Zeros | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| All Zeros | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| MPA CRC | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Figure 7.  SendSE Message Containing an iSER HelloReply Message 
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iWARP Message Format for SCSI Read Command PDU 


The following figure depicts a SCSI Read Command PDU embedded in an 
iSER Message encapsulated in an iWARP SendSE Message. For this 
particular example, in the iSER header, the Write STag Valid flag is 
set to zero, the Read STag Valid flag is set to one, the Write STag 
field is set to all zeros, and the Read STag field contains a valid 
Read STag. 


0 1 2 3 
01 .2.394.5.6. 71.8.9 0 L:2.3-—4 5-6 T 8 9.0.1. 2 3 4 5-6 7.8.9 0 1 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| MPA Header | DDP Control | RDMA Control | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| Reserved | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
| (Send) Queue Number | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 

| (Send) Message Sequence Number 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
(Send) Message Offset | 
一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
0001b [011| All zeros | 
一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
All Zeros | 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
Read STag | 

十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
SCSI Read Command PDU | 

/ // 


一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
MPA CRC | 


| 
+ 
| 
+ 
| 
+ 
| 
+ 
| 
/ 
| 
+ 
| 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Figure 8. SendSE Message Containing a SCSI Read Command PDU 
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The following figure depicts an iWARP RDMA Write Message carrying 
SCSI Read data in the payload: 


0 


I 2 


3 


Q 1-2; 3475.65 :0° 2 2 :3* 4. -5- 677 58.90 1 253: 4 75:6 703892 Oe 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 


et al. 


十 


十 


MPA 
一 十 一 十 一 十 一 十 


一 十 一 十 一 十 一 十 


Header | DDP Control 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
Data Sink STad 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 

Data Sink Tagged Offset 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
SCSI Read data 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
MPA CRC 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


十 一 一 


十 


十 


一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
RDMA Control | 
一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 

十 
一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 

// 


一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


RDMA Write Message Containing SCSI Read Data 


Standards Track 


[Page 76] 


RFC 5046 iSER Specification October 2007 


A.5. 


Ko, 
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The following figure depicts a SCSI Write Command PDU embedded in an 
iSER Message encapsulated in an iWARP SendSE Message. For this 
particular example, in the iSER header, the Write STag Valid flag is 
set to one, the Read STag Valid flag is set to zero, the Write STag 
field contains a valid Write STag, and the Read STag field is set to 
all zeros since it is not used. 


0 T 2 3 
0L 2:34 ».6.1.8.9 0 1:2.3-—4 5-6 7 8 9 0 1.2 3 4 56 1.9.9.0 t 
T-—-—tR—R-—t-tL-—R-.-L-R—x—L—4—4——4- BB BM B--4---4--4 
| MPA Header | DDP Control | RDMA Control | 
T—k-—R—R-—t-tL-—k-x-L-4-—4-—L—4-—4——4—x-——4---4--—4-4--4-4--4--4 
| Reserved | 
T—-—tR-—R-—-—L-—R-.-L-—R-—-—L—4—x-—4--—4-4-L-4-BM—B-B——--4--4 
| (Send) Queue Number | 
K A E A A L L A L ee R 44-44-44 

| (Send) Message Sequence Number 
T—t-—R-—R-—-tL-—R-.-L-—R-—x-L-4-—4-——4-x4-—4-—x--4-4--4-4--4---—4--—4 
(Send) Message Offset | 
Pp i mn he tsten tektar ee eaea ekta aee 
0001b |1|0| All zeros | 
一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
Write STag | 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
All Zeros | 

十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
SCSI Write Command PDU | 

/ // 


一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
MPA CRC | 


| 
+ 
| 
+ 
| 
+ 
| 
+ 
| 
/ 
| 
+ 
| 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Figure 10. SendSE Message Containing a SCSI Write Command PDU 
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An iSCSI R2T is transformed into an iWARP RDMA Read Request Message. 
The following figure depicts an iWARP RDMA Read Request Message: 


0 


I 2 


3 


O12: 345. 65 :0 9 2 5:2 :3' 4. -5- 6:7 78. 9:0. 1 2, 3: 75:6 :1::87 97 0. X 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
MPA Header | DDP Control | RDMA Control | 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Reserved (Not Used) 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


DDP 


(RDMA Read Request) Queue Number 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


DDP (RDMA Read Request) 


Message Sequence Number | 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


DDP 


(RDMA Read Request) Message Offset 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Data Sink STag (SinkSTag) 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


| 
+ 
| 
十 一 
| 
十 一 
| 
十 一 
| 
+ 
| 
十 一 
| 
十 一 


eta 


Data Sink Tagged Offset (SinkTO) 


| 
+ 
| 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


RDMA Read Message Size (RDMARDSZ) 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Data Source STag (SrcSTag) 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Data Source Tagged Offset (SrcTO) 


| 
+ 
| 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


MPA CRC 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Figure 11. RDMA Read Request Message 
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iWARP Message Format for Solicited SCSI Write Data 


The following figure depicts an iWARP RDMA Read Response Message 
carrying the solicited SCSI Write data in the payload: 


0 


1 2 


3 


Q 1-2; 3475.65 4:8) 190s 2 :3* 4. -5- 67178. 9 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


十 


十 


+-+- 


+-+- 


Figure 12. 


et al. 


MPA Header | DDP Control 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 
Data Sink STad 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 
Data Sink Tagged Offset 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 
SCSI Write Data 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 
MPA CRC 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 


十 


十 


十 


十 


十 


RDMA Control | 
一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


十 


一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


| 
// 


一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


RDMA Read Response Message Containing SCSI Write Data 
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A.8. iWARP Message Format for SCSI Response PDU 


The following figure depicts a SCSI Response PDU embedded in an iSER 
Message encapsulated in an iWARP SendInvSE Message: 


0 1 2 3 
012345678290123456789012345678901 
T—-—R-—R-—-—L-—R--L-R-—-—L-4-—x4-—L— 4-4 rS i ste De ee i et N E E EA 
| MPA Header | DDP Control | RDMA Control | 
A Eh ts CD Es G R De dy en Oy ey ty C G L 
| Invalidate STag | 
T—-—tR—R-—-L-—R-x-L-—4-—x-—L—4-—4——4-—x-—4--—4--—4-4--4-4--4--9 
| (Send) Queue Number | 
T—-—R-—R-—-tL-—R-.-L-—4-x4-L-4—4——4—4——4-44-BBMB—B—-.-.--4--4 

| (Send) Message Sequence Number 
K L a H a G L S LE G G V G C G 44MM BM—B--4-4--4-4--4 
(Send) Message Offset | 
a G S E G a H R L A A L S C L G a A EN 
0001b 1010) All Zeros | 
a N a E S G A H T K H C N et C G A L V B-6 
All Zeros | 


十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 

All Zeros | 

十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
SCSI Response PDU 

/ // 


一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 
MPA CRC | 


| 
+ 
| 
+ 
| 
+ 
| 
+ 
| 
/ 
| 
+ 
| 
十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 一 十 


Figure 13. SendInvSE Message Containing SCSI Response PDU 
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Appendix B. Architectural Discussion of iSER over InfiniBand 


B.1. 


Ko, 


This section explains how an InfiniBand network (with Gateways) would 
be structured. It is informational only and is intended to provide 
insight on how iSER is used in an InfiniBand environment. 


The Host Side of the iSCSI and iSER Connections in InfiniBand 


Figure 14 defines the topologies in which iSCSI and iSER will be able 
to operate on an InfiniBand Network. 


十 一 一 一 十 一 十 一 一 一 十 十 一 一 一 十 一 十 一 一 一 十 十 一 一 一 十 一 十 一 一 一 十 十 一 一 一 十 一 十 一 一 一 十 十 一 一 一 十 一 十 一 一 一 十 
|HcA| |HcA| |HCA| |HcA| |HcA| |HcA| |HcA| |HcA| |HCA| |HCA| 
*-V— t-v-+ t+-v-+ t-v-+ t+-v-+ t-v-+ t-v-+ t-v-+ t+-v-+ 4+-v-+ 


| ----+------ | ----- + 一 一 一 一 一 |----- 4d----- |----- + 一 一 一 一 一 | ----- +---> To IB 
IB| IB | IB | IB | IB | SubNet2 SWTCH 
qe E l OE ECCE Macc luca em Rd COM Monum 十 
| InfiniBand Switch for Subnetl 
+---+----- +-------- + 一 一 一 一 一 + 一 一 一 一 一 一 一 一 + 一 一 一 一 一 + 一 一 一 一 一 一 一 一 一 一 一 一 v------ + 
| TCA | | TCA | | TCA | | 
4£----- 十 + 一 一 一 一 一 十 4----- 十 | IB 
/ IB UX / IB 7X / \ 十 一 一 十 一 一 VV 一 一 十 一 一 十 
| isER | | isER | | IPoIB | LEGS il 
| Gateway | | Gateway | | Gateway | 中 三 三 一 二 二 十 
to to to Storage 
| iSCSI | | iSER | | IP | Controller 
| Tce | | iWARP | [Ethernet | + 一 一 一 一 一 + 一 一 一 一 一 + 
+ 一 一 一 V 一 一 一 一 一 | + 一 一 一 V 一 一 一 一 一 | 4£----y----4 
| EN | EN | EN 
me 站 二 +----> to IP based storage 


Ethernet links that carry iSCSI or iWARP 
Figure 14. iSCSI and iSER on IB 


In Figure 14, the Host systems are connected via the InfiniBand Host 
Channel Adapters (HCAs) to the InfiniBand links. With the use of IB 
switch(es), the InfiniBand links connect the HCA to InfiniBand Target 
Channel Adapters (TCAs) located in gateways or Storage Controllers. 
An iSER-capable IB-IP Gateway converts the iSER Messages encapsulated 
in IB protocols to either standard iSCSI, or iSER Messages for iWARP. 
An [IPoIB] Gateway converts the InfiniBand [IPoIB] protocol to IP 
protocol, and in the iSCSI case, permits iSCSI to be operated on an 
IB Network between the Hosts and the [IPoIB] Gateway. 
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B.2. The Storage Side of the iSCSI and iSER Mixed Network Environment 


Figure 15 shows a storage controller that has three different portal 
groups: one supporting only iSCSI (TPG-4), one supporting iSER/iWARP 
or iSCSI (TPG-2), and one supporting iSER/IB (TPG-1). 


+--+--v--+----------+--v--+----------+--v--+--+ 
| | IB | | iWARP | | EN | | 
| | | | TCP | | NIC | 
| | (TCA) | | RNIC| | | | 
| 0----- | + 一 一 一 一 一 + + 一 一 一 一 一 + | 
| TPG-1 TPG-2 TPG-4 | 
|| 9 E 33 9.1.2.4 9.1.2.6 | 


Figure 15. Storage Controller with TCP, iWARP, and IB Connections 


The normal iSCSI portal group advertising processes (via the Service 
Location Protocol (SLP), the Internet Storage Name Service (iSNS), or 
SendTargets) are available to a Storage Controller. 


B.3. Discovery Processes for an InfiniBand Host 


An InfiniBand Host system can gather portal group IP addresses from 
SLP, iSNS, or the SendTargets discovery processes by using TCP/IP via 
[IPoIB]. After obtaining one or more remote portal IP addresses, the 
Initiator uses the standard IP mechanisms to resolve the IP address 
to a local outgoing interface and the destination hardware address 
(Ethernet MAC or IB GID of the target or a gateway leading to the 
target). If the resolved interface is an [IPoIB] network interface, 
then the target portal can be reached through an InfiniBand fabric. 
In this case, the Initiator can establish an iSCSI/TCP or iSCSI/iSER 
session with the Target over that InfiniBand interface, using the 
Hardware Address (InfiniBand GID) obtained through the standard 
Address Resolution (ARP) processes. 


If more than one IP address is obtained through the discovery 
process, the Initiator should select a Target IP address that is on 


the same IP subnet as the Initiator, if one exists. This will avoid 
a potential overhead of going through a gateway when a direct path 
exists. 
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In addition, a user can configure manual static IP route entries if a 
particular path to the target is preferred. 


B.4. IBTA Connection Specifications 


The InfiniBand Trade Association (IBTA) connection specifications are 
outside the scope of this document, but it is expected that the IBTA 
has or will define: 


* The iSER ServiceID. 


* A Means for permitting a Host to establish a connection with a 
peer InfiniBand end-node, and to fall back to iSCSI/TCP over 
[IPoIB] if that peer indicates iSER is not supported. 


* A Means for permitting the Host to establish connections with IB 
iSER connections on storage controllers or IB iSER connected 
Gateways in preference to [IPOIB] connected Gateways/Bridges or 
connections to Target Storage Controllers that also accept iSCSI 
via [IPoIB]. 


* A Means for combining the IB ServiceID for iSER and the IP port 
number such that the IB Host can use normal IB connection 
processes, yet ensure that the iSER target peer can actually 
connect to the required IP port number. 
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